From patchwork Mon Jan 9 20:52:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094267 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA9A2C6379F for ; Mon, 9 Jan 2023 20:53:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0978F8E0005; Mon, 9 Jan 2023 15:53:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 048058E0001; Mon, 9 Jan 2023 15:53:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2B108E0005; Mon, 9 Jan 2023 15:53:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D39AB8E0001 for ; Mon, 9 Jan 2023 15:53:46 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id ACF0FA07C4 for ; Mon, 9 Jan 2023 20:53:46 +0000 (UTC) X-FDA: 80336462052.06.CC2A054 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf14.hostedemail.com (Postfix) with ESMTP id 1874C10000D for ; Mon, 9 Jan 2023 20:53:43 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hV1eYOET; spf=pass (imf14.hostedemail.com: domain of 31368YwYKCOoegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31368YwYKCOoegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NqXJ+ZomBW9TS34c/jGDj4kdswhUlT56T9j0TeoADNk=; b=1Amb53ZOC9uktrl/yr1QbrLeYlRafzV7dENdXZ1NyjQXqoGjZNSTmlYJeBMcyfm3ye9GpG 3v2pt2JLVJRsPj6tkDJWH16lHFY87eqp05tDqZsPxLrrKviZ6YKi4cNR4ovNovlkgtLr+r msJdAm3aikwUenIzmmV86WtED3CtAbA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hV1eYOET; spf=pass (imf14.hostedemail.com: domain of 31368YwYKCOoegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=31368YwYKCOoegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297624; a=rsa-sha256; cv=none; b=I5kOl63PWSQVCXGaXuyyc1ogUOzyiWIIozaatrOhoS1ER7SC93o0Pp+PDg0kLY0kaXeLFC lzlJ5lp9gCYyijzjwMtT2oO2fk/P1KmojSuq19FBMAfC/I7OziiZDUG8XclR3DWkbX8GpZ DRd3I/S0eHxhiNoTmJhyNqb4LBALIK0= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-4c36d00c389so91826167b3.18 for ; Mon, 09 Jan 2023 12:53:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NqXJ+ZomBW9TS34c/jGDj4kdswhUlT56T9j0TeoADNk=; b=hV1eYOET6T2afidyHxuU+TDalPYH7SoLHvYl4BYwO2+JPtGGpug5jlQUoy9K4emxnp wkRJchtDj68k/F+2XFNh7yn/DcVU/S6b5BfTlEf/HmlwKUP7B8xL+hIWlClC+stgjn0Q yjEqU4w0bHW24Tkvl4HAvFsvboqjzGC/OqOppLTxN2NoinKdWAqCYIk7Wp05H1K9TySk fMjEc/Qrcjc9huA2llMuqioo8fVnZzgiNBMYUce/o/EkMO+/Qtv30xUsop9ErVhi9UD1 18fRcEvtN38NaeuSdMu7DJkgG+1MjHvnJbjraNd54FiH0BDnP0iYXASfyl6JxE09JvbK zNVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NqXJ+ZomBW9TS34c/jGDj4kdswhUlT56T9j0TeoADNk=; b=wvWaVdjS28Fnx82RC6Ln6bI7z/TqhVQ+sQjwJymy//dk/qIbBNCPrKYeNcWyEI9blE 1IRFdChOHk8zKFM9KsU4WifUFo7oTjpWU1GBvVKEN2UhuhzABoZ8sCYAcF9HpVIv3Ll+ QgDZPwDwpJlm46CoKk11uskmU/4//9BM9QUuFiFNqGcLeBTu0Vl6GKQvEaCeUnBtyhW/ RcBvd10y76AofbBsDfCzo9tskVjgZcH/i4JVBq9/pEzBumd+dAtnQ8rrtxIXMg9RvnQw VG+fLf78PSe/QVRfYzF0At92I4xDJolphsHVLZRrIDwothvHWiDYDDiDC6pfrLtY0GV0 YchQ== X-Gm-Message-State: AFqh2koW3oYa+ZEz6rbbqpHGxzIWtyskujCFo7V1vru0KvRI16kQ7Sau EVNrf8gj0LEN6iTivoz6/gG984g32BY= X-Google-Smtp-Source: AMrXdXsc+lRNNMUj/grXDfNh64QUbjHDxIxAaFKRrkqMjOPH6SvLaEBts0d+5j6B8x6v1ECq7EPEJR82hgY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:7189:0:b0:707:18f:7226 with SMTP id m131-20020a257189000000b00707018f7226mr210074ybc.505.1673297623117; Mon, 09 Jan 2023 12:53:43 -0800 (PST) Date: Mon, 9 Jan 2023 12:52:56 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-2-surenb@google.com> Subject: [PATCH 01/41] maple_tree: Be more cautious about dead nodes From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Stat-Signature: cg1hh3muccyenpohrz7j8ojbdfaq7myk X-Rspam-User: X-Rspamd-Queue-Id: 1874C10000D X-Rspamd-Server: rspam06 X-HE-Tag: 1673297623-688490 X-HE-Meta: U2FsdGVkX18kSq3Et5wSEAiZDMOhxN7ZUXfWwPrIxXj/XMti8YyoAlUPGic0Jpy9lVdZCyBZXo85U/8LkqA3SaPljYbCD9mIdfjdr6yrQwjhAA7wfdinalhfjOZJ2ZbVp9WJNMy4PA0ov6MkfEFR0P5MWmHRzc7S/J/sqE6ikpRw/pXIqTsHPElkO4UFk0qBbKdmAPoEtb/1ASbNe8DpS+Xp096ksmBlJ9wm0grtYpvpkNdtD3awD6q9zdWrUfs9nk/calkDw7SnAuZffGpnET3eAmGfgmXIn+R6D1XhLBiiOgaVgTI9ZnQ841uY0W9zZwuGNr4LdSTAgztEgH2mjBavZ9Fp6AJ5lRjm/nEwqt9KfhO1SE6tSliB8PklE4DNcekqhbxhQ7Fk6nad8pI4d01HdNFsV2ZLIWnsDPWjz8uR6+ysSxyskRcBQgX4nTxaQPheSL9uObyFuFkCMLd80TaVvxssoTlUX/NsoIDl6t38FELRZirdc6tOOSRbHYJsxrbSqRX3/DPBBTJ7Y14SPJz0K3Zr56zG51EQbiVynIeK8jFOwcBt53q8qjw3KKHNN3RG8dx989r1LW858tNTjErWxJdEb3xYygVtLaOIc2CD1NgZ+WqfIDdfYefrlX6qOMHiLzSCPMdn5yJdPypRJmBz7OKUC7qdgyvXXHWxXNmMSdKpRtsYZucZzgZ/BvcKySRGhJuZqAR3uTyTsKZE+Z47/aQoTHJMECGx51d3yLU+Oyfm1nNaaLVQlvz67NB2t4KZgs+0hNwm8/L2ysm+CEl+CX/2Vw5STQ5GMqL9HGa2VDvp4sKM8lGfyNc180hKIf2SfIfEeKW1v5Y1OS0YD0TdeIjf5ygOBZjt13e9x2SpQ/jXYsxTDXiI1xXUA0HO3zXHoX2UbwGd9xVqQ7q/zW1u0307Dd37sF4AntK31/0avXwOGkXNTIosgQVegI1wq47/b0UsQ0Qd5ubZWsT 571TqmsY Qp0Ut2T1KNJgIhcul2q2rlUg1bp9lbOcGVcXl252pA70YG1MAY9cUXl2STuWLGF+wPu1IObL+wKONIb1Xz8z7l8+14eV2i/CvREgwqHY79jW8yzj5mXSZ4n4gCHI0ONdX355rYErWukZwFuqQxGaenZLOqdwpevJRcBV4iiUp8hOneo6Au0wzQCRjmGB0ySVnjmJnhr4wLvzOb9k5MEv025PAepIiKGiGUlypslOaCwpVGpVLfkLjTyZFxZ6Gnfjp75Jr7KdWhsMogoTJE5DgRd9GwiYxKKfZY8jLx98DZp1w2UesEu5To4KL4SM6x4nLMDlJmGLpIenkB4e26P6vmXUkWOj58Rc4tVTeAdaaKiifedSi7aarvERkCTI4+udasOZGVZ+Xg61VTYkZzL+ii6A1vu9P81rsIYwF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett ma_pivots() and ma_data_end() may be called with a dead node. Ensure to that the node isn't dead before using the returned values. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 53 +++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 26e2045d3cda..ff9f04e0150d 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -540,6 +540,7 @@ static inline bool ma_dead_node(const struct maple_node *node) return (parent == node); } + /* * mte_dead_node() - check if the @enode is dead. * @enode: The encoded maple node @@ -621,6 +622,8 @@ static inline unsigned int mas_alloc_req(const struct ma_state *mas) * @node - the maple node * @type - the node type * + * In the event of a dead node, this array may be %NULL + * * Return: A pointer to the maple node pivots */ static inline unsigned long *ma_pivots(struct maple_node *node, @@ -1091,8 +1094,11 @@ static int mas_ascend(struct ma_state *mas) a_type = mas_parent_enum(mas, p_enode); a_node = mte_parent(p_enode); a_slot = mte_parent_slot(p_enode); - pivots = ma_pivots(a_node, a_type); a_enode = mt_mk_node(a_node, a_type); + pivots = ma_pivots(a_node, a_type); + + if (unlikely(ma_dead_node(a_node))) + return 1; if (!set_min && a_slot) { set_min = true; @@ -1398,6 +1404,9 @@ static inline unsigned char ma_data_end(struct maple_node *node, { unsigned char offset; + if (!pivots) + return 0; + if (type == maple_arange_64) return ma_meta_end(node, type); @@ -1433,6 +1442,9 @@ static inline unsigned char mas_data_end(struct ma_state *mas) return ma_meta_end(node, type); pivots = ma_pivots(node, type); + if (unlikely(ma_dead_node(node))) + return 0; + offset = mt_pivots[type] - 1; if (likely(!pivots[offset])) return ma_meta_end(node, type); @@ -4504,6 +4516,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) node = mas_mn(mas); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + mas->max = pivots[offset]; if (offset) mas->min = pivots[offset - 1] + 1; @@ -4525,6 +4540,9 @@ static inline int mas_prev_node(struct ma_state *mas, unsigned long min) slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); offset = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + if (offset) mas->min = pivots[offset - 1] + 1; @@ -4573,6 +4591,7 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, struct maple_enode *enode; int level = 0; unsigned char offset; + unsigned char node_end; enum maple_type mt; void __rcu **slots; @@ -4596,7 +4615,11 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, node = mas_mn(mas); mt = mte_node_type(mas->node); pivots = ma_pivots(node, mt); - } while (unlikely(offset == ma_data_end(node, mt, pivots, mas->max))); + node_end = ma_data_end(node, mt, pivots, mas->max); + if (unlikely(ma_dead_node(node))) + return 1; + + } while (unlikely(offset == node_end)); slots = ma_slots(node, mt); pivot = mas_safe_pivot(mas, pivots, ++offset, mt); @@ -4612,6 +4635,9 @@ static inline int mas_next_node(struct ma_state *mas, struct maple_node *node, mt = mte_node_type(mas->node); slots = ma_slots(node, mt); pivots = ma_pivots(node, mt); + if (unlikely(ma_dead_node(node))) + return 1; + offset = 0; pivot = pivots[0]; } @@ -4658,16 +4684,18 @@ static inline void *mas_next_nentry(struct ma_state *mas, return NULL; } - pivots = ma_pivots(node, type); slots = ma_slots(node, type); - mas->index = mas_safe_min(mas, pivots, mas->offset); - if (ma_dead_node(node)) + pivots = ma_pivots(node, type); + count = ma_data_end(node, type, pivots, mas->max); + if (unlikely(ma_dead_node(node))) return NULL; + mas->index = mas_safe_min(mas, pivots, mas->offset); + if (unlikely(ma_dead_node(node))) + return NULL; if (mas->index > max) return NULL; - count = ma_data_end(node, type, pivots, mas->max); if (mas->offset > count) return NULL; @@ -4815,6 +4843,11 @@ static inline void *mas_prev_nentry(struct ma_state *mas, unsigned long limit, slots = ma_slots(mn, mt); pivots = ma_pivots(mn, mt); + if (unlikely(ma_dead_node(mn))) { + mas_rewalk(mas, index); + goto retry; + } + if (offset == mt_pivots[mt]) pivot = mas->max; else @@ -6613,11 +6646,11 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, while (likely(!ma_is_leaf(mt))) { MT_BUG_ON(mas->tree, mte_dead_node(mas->node)); slots = ma_slots(mn, mt); - pivots = ma_pivots(mn, mt); - max = pivots[0]; entry = mas_slot(mas, slots, 0); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + max = pivots[0]; mas->node = entry; mn = mas_mn(mas); mt = mte_node_type(mas->node); @@ -6637,13 +6670,13 @@ static inline void *mas_first_entry(struct ma_state *mas, struct maple_node *mn, if (likely(entry)) return entry; - pivots = ma_pivots(mn, mt); - mas->index = pivots[0] + 1; mas->offset = 1; entry = mas_slot(mas, slots, 1); + pivots = ma_pivots(mn, mt); if (unlikely(ma_dead_node(mn))) return NULL; + mas->index = pivots[0] + 1; if (mas->index > limit) goto none; From patchwork Mon Jan 9 20:52:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094268 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80963C5479D for ; Mon, 9 Jan 2023 20:53:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A9458E0006; Mon, 9 Jan 2023 15:53:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 132AD8E0001; Mon, 9 Jan 2023 15:53:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3D058E0006; Mon, 9 Jan 2023 15:53:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E60228E0001 for ; Mon, 9 Jan 2023 15:53:48 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B12AC120BD5 for ; Mon, 9 Jan 2023 20:53:48 +0000 (UTC) X-FDA: 80336462136.22.B74E207 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf12.hostedemail.com (Postfix) with ESMTP id F2EB94000E for ; Mon, 9 Jan 2023 20:53:46 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cDmwrsTn; spf=pass (imf12.hostedemail.com: domain of 32X68YwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=32X68YwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297627; a=rsa-sha256; cv=none; b=LZh0yNrmp3jLt3/Bb/n+A/tfvMZ7JOs0R/uQEN7asgF23xWSrQWZiFz0R1YniKYQoPp5wM zcmfEZrOUHXYsZjGmRL7ULE3Ihd44Nr2NvcFxZAEM4UPZx1zfa4HuOBXnAAtTai3vBxrWr OA38UAOLoS/PwZTOzfkfmkZ+g0UXWeM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=cDmwrsTn; spf=pass (imf12.hostedemail.com: domain of 32X68YwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=32X68YwYKCOwgifSbPUccUZS.QcaZWbil-aaYjOQY.cfU@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297627; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GU7SFbwLvAPdD9ofPJY8ZC3AFrLWlaqOvHr/TBsM1Ak=; b=H8rFJi51AgRruzSpe1fTDQpO208KygKZe2XJd7nus0n8TaRM60BtGAKni6KgKwxWSztKjW k7Vv66n3jcj0aztuljGyyHBCfS1JPVD/tsaW5vublLEhKTTx6BPSJF4qLJL/htnpTGngLX M6G/uqlApa1SsuF11Piopv1MWY5kqMo= Received: by mail-pl1-f201.google.com with SMTP id l15-20020a170903244f00b001927c3a0055so6949251pls.6 for ; Mon, 09 Jan 2023 12:53:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GU7SFbwLvAPdD9ofPJY8ZC3AFrLWlaqOvHr/TBsM1Ak=; b=cDmwrsTnYnxMRe2M80E4L23YmooQix5wB2D0eUb+oN2Hp3bbyg7HFQB/PgVNLoHyKT pK0aWtF4CTk14CmlPxG5u4iqhHXIU+wNQHLdGD94tpFP5D5/s9ok9CC5dVKy7ZTVeaNC t3/Bp5EGifXzniBaXl20JHyzpYwXEFHr6vzZDBUr27QQthKbHcJnIlWiGRlMM6UQg0ND gIbAYHo5+gFS5Z1JXyH/8Q0hICWyuNm77sn6kp0Bt2Cy3fEKAr803t0EnzzgxBkOELQu v2Q/SfqHP/lVu/l9N+v6WarwdrrXQ25UydUZpK32X5OyRK1NLw7E4rGZtvAQp6+5paD/ Xoog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GU7SFbwLvAPdD9ofPJY8ZC3AFrLWlaqOvHr/TBsM1Ak=; b=OlFGqmQRVY+yp3uIrZdjSjJCu+FgJAlgrhqy0Q88zAZ7LJKhigSFiKR8V4Qs137yRF Cc9zUIMq2+SQ9/qqzgS2S/EPGnHZV+zoZInsnqwO1jvou7ocCdHggXE9TFUxcRRfYHRJ oji81DfiwCXEqVxFL12kF62o5JaV+jQXe9RqqGCeNI3QaxYRmVizb+U7Wa6yrrjuLwoh Q93qqGHIo9yu/TOM2EribUlrrXsSqA6oCu7SSMTR23TWccJPDtsQC9PEsHMe675zePrT QDjZKU9zr+HxrsaXQT04Gde8qui/MSzn+5p4WYRVkfOCrjjiHB2ca/65jTKn+4gonUbO PgMw== X-Gm-Message-State: AFqh2krwZBprQP0O2oDaI9dhW+waYuzRS2CtXlDDFe7CNwvO/E2V0qA4 2nRDFeBc6rB9C/K/p180sHDOjqmsH3c= X-Google-Smtp-Source: AMrXdXsx8jOEqF8lBOHYbim74eUIuqk/BpJXbormK1OrkMgDb8c5pT4kOfmHApmrR7R22KnvvmwfjroHm1I= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:902:c209:b0:177:faf5:58c5 with SMTP id 9-20020a170902c20900b00177faf558c5mr4095626pll.166.1673297625715; Mon, 09 Jan 2023 12:53:45 -0800 (PST) Date: Mon, 9 Jan 2023 12:52:57 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-3-surenb@google.com> Subject: [PATCH 02/41] maple_tree: Detect dead nodes in mas_start() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Rspam-User: X-Rspamd-Queue-Id: F2EB94000E X-Rspamd-Server: rspam01 X-Stat-Signature: o1jewnsugkk8aiaopakemhc59eu9gknw X-HE-Tag: 1673297626-789174 X-HE-Meta: U2FsdGVkX195cVYtq0WK05o4MtQw6v7CiaKRu7CgTsBXZHUCa4pPITNnKUZWqPPXDScvVHnfVNL4QCtHWXiNuQWRXn0iJ5nfJDKP/isShsdSkkAU23zWXf/tcFAZygNLJaQqKwpCC0QcqgXrT87F3AAg7ahevRpj3Z7vPsSzC4zw+4MxvDjL4//C4g6pTX1eCUUQgHgnPRQHk0DMfvSkjbKuTiNiRfsgxgiUCgoUsIzJqSs0Fds4NCvXcvqD/UyIOzN0q4za6uftLI6WhNf9xDhZWjOKl9T3HuFoZwm+uOxMQjJB9wi8HhH8xSl0HqBMGZy+JIljZrp9N8kN+M9YHIe45fKLkyrUZ6AbCFpufeD2axDeCOLyr0ZJ1ffIn1CGD15ZRmMO50828cFLuIyq1Iq1VRX9C39JrP2Oze7ApQTsO8rps+Ed1oa/a3h9sCjR3qYXGsTvJXR3eflwnRXoVxyonforQbYL2J4RCf4ihMrCry4XPCIErSdfXl0j+TtVWsGFUBaM4iAN81UQieONYHotj8woP4mKR/5MtVL6/UvQDw0gAQIcyWNNPxwvTmWBbyKMmVNhHKNuFLHTSH0h4WNt/qkb5xlooerOUxXUa2B0PhpGoIMaQoW80ON5njWOhW8LQJIH+MutDQoNN/zMwHFDo/KRqzalHmJSWyiizbXdfZbqpvc54SBwMfv5p9DUG8Lo0eYWHghNJ9bkBSikn+HSbkajhU1GjduxgZ28TXl/IhRhC3+VgZcarKQRm8muiK08ECOi2yKN9r375xBIZfGWqksMZWUQa8Uoc5Vye7NA8Z1qJIdiHcG22udkm2YUnMZ9w6bAmbwBi3kC1wdL7hWQFgyCZfts6T3FISylP/qxIxmhBfq2frIv1pb4Zir7xjXEtWLl8uhShJU1yUyEYdVDk9Y5BId5EkO5100ayfndOX06xeMkbm+IYg9uVmnHQAHZjjaUJ1WcG542vVZ QBBW7dzI kykR7u9oy6MVGytAlg+oOYDleciuItDKSnogMw4x0P+hje3DQZ1PffBIWWl4RjT0iWw3Z+BGDacqqH0Fb1lz4uxmsuI8bbtNicWoLxoVycv/w0jTBBixznPZbTDyEiVPCnA/WU7Vmky6x92FQE8kLg6ZSOjHFs53giYCzjWTK9Pqij/t+CEMPYXKdOEbogVhnUmwiThofNrkmG020qe6NRV7SLIg2wpWUMV1oYcdbfSNYC7X/Lon93JKdO/+EuHFghSlw+ZgdWPLD/+d0JEiXYmhrgEE8BsAgVS13kzIYa3aboZNbpAtSuwCZIgYdGk7IE+9B5POTivOrncrG8N+pHiFyQfjx/5GYhIurznOWNEfuE8aT82Bn7SZzSPBs6nFQWeBKYEeR0toeHWOZP7Mcd+flv9NQF4HOVZsH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett When initially starting a search, the root node may already be in the process of being replaced in RCU mode. Detect and restart the walk if this is the case. This is necessary for RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index ff9f04e0150d..a748938ad2e9 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -1359,11 +1359,15 @@ static inline struct maple_enode *mas_start(struct ma_state *mas) mas->depth = 0; mas->offset = 0; +retry: root = mas_root(mas); /* Tree with nodes */ if (likely(xa_is_node(root))) { mas->depth = 1; mas->node = mte_safe_root(root); + if (mte_dead_node(mas->node)) + goto retry; + return NULL; } From patchwork Mon Jan 9 20:52:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094269 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE90DC54EBD for ; Mon, 9 Jan 2023 20:53:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41A258E0007; Mon, 9 Jan 2023 15:53:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A31A8E0001; Mon, 9 Jan 2023 15:53:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F5AE8E0007; Mon, 9 Jan 2023 15:53:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0F6A38E0001 for ; Mon, 9 Jan 2023 15:53:51 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id AB6EB140C10 for ; Mon, 9 Jan 2023 20:53:50 +0000 (UTC) X-FDA: 80336462220.08.44AE2DA Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 0DF63180007 for ; Mon, 9 Jan 2023 20:53:48 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="nFEf7/Su"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 33H68YwYKCO8jliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=33H68YwYKCO8jliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297629; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tYS2Mdxn+jwv2Pf7FZ7D7lMnoZhtkSPqAfmT0KRte/A=; b=3eN4wKDcecioKR4wRxdl+N8XMCYskT4HZuHbEsRH/YdnU/qw4uI1uhTnf3jILhZSR9kO1v X+pF1CA5psrYWW/2SThPYRzNQwKJCD2YIqr/xu3zMsxiqNG6DGCDv4lRvrLUF9wgj1D4RX QppUDjW/j0VItiU7aLp8q33OfyBnJak= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="nFEf7/Su"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf06.hostedemail.com: domain of 33H68YwYKCO8jliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=33H68YwYKCO8jliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297629; a=rsa-sha256; cv=none; b=UKK7sbjDCkGzl03iDgChfnwkIKfV0blcInS8oJfQm5+sn6uws4hkIiYPuEEmwZb8VuYrlJ YjxCOda6gsc6ZFZMqfx2YXv0iIY+2yrK9zgq99cGxsZWLNw2a9SXKWR2U4u5Vq2BCPEqi1 xUMzoS+8zA7gNC4LNbmzVSN+i8wMzlY= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-4c11ae6ab25so104120347b3.8 for ; Mon, 09 Jan 2023 12:53:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tYS2Mdxn+jwv2Pf7FZ7D7lMnoZhtkSPqAfmT0KRte/A=; b=nFEf7/SuzvGfFfBp8alBHLcE/S20BVhkxOFCKJJxDEEpkBCqnPrEjXLQRPhpajIyrV J/MheNxTRWGkew5780i2M8uqtnyvNe0zoFPa24hnhtsW7xvdNOQ3PiEyoeTmA3WWPiUh QK2oHtldPphM+TCue7rjkADeiZ4F/aOO5f9+1p0Hswrqrn/9/CHXJyrN6vRAUq5GDqS7 Axk6TqH0bcf1b4jM3d3LsItdyw7HXfIeFcgC6ZYFCu5leTcjZd3mHlVr8yfm4xVeOVbq OV6UpEzwnJ3oSH64LbnS9rS633sT7MF5+6p/WT9/4pbYtB4HJqqA4EN1qZDYnvdMkGJo Aq4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tYS2Mdxn+jwv2Pf7FZ7D7lMnoZhtkSPqAfmT0KRte/A=; b=Xg6RxOtJLlqsHvoXPHCGB45lwH15w9JWcLOYoJ0a2OG7hMRWFyg3swEFPYx1pwbXLF G2fwgZs+j41QTniiBcTsu9PxElhI/wvoSlV/8mJJti1ue6w2kqcOp6GOXBVJhgqcG61V vETHVNNIdXUIlpiwxunHpFpzmtAc0kfx0XBxIPDbzZ8ZyDF2vcLCrZDoammDN9wTfIUm df7mTQA7kriSoGrUAjaJ3BCCOAHhaLM7wDoWNl84R6qV6AqgrHRo/XogOpQ8u8jgg4KN 8mk02AFdIqP78It9Zvwa6ToB/KjeCM/Y0GMoYeFRPIxtEjlZQxelkgDfTctCPpY1xGCM 22cw== X-Gm-Message-State: AFqh2kroO3J5/1KVITCI5Spf7ZEK/M8P4SuixDKvWkIIg89pMaCgGFP4 gXmWSl3lApJJkVCDe2ahq0JuR/U1I3k= X-Google-Smtp-Source: AMrXdXs7OcQqihjg4pdS+IdnJHYFJuvOEzRgXy7ctwOn05/Em3c821T3ojLTZvqk1LIxMtAuoTuisCHM2Gs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:dd07:0:b0:7b4:db9a:48ae with SMTP id u7-20020a25dd07000000b007b4db9a48aemr1234542ybg.207.1673297628223; Mon, 09 Jan 2023 12:53:48 -0800 (PST) Date: Mon, 9 Jan 2023 12:52:58 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-4-surenb@google.com> Subject: [PATCH 03/41] maple_tree: Fix freeing of nodes in rcu mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Rspamd-Queue-Id: 0DF63180007 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: xfhs58qxgy5yjiphj3164pxuuyj3rnbu X-HE-Tag: 1673297628-614989 X-HE-Meta: U2FsdGVkX19qtXcu2CSQAkFCG0UO97W0gJx09MlDbqfqx9Q8BxMewKT3M7t24gut+MhdpBPP83gmMXLZAghhnfbpy8xLkL77Wvn4YE7JwELRVvW1oDviI2d5iI4ifT+rbwuhleRrEhl07oboY+LNL72pVteI1kZcQ2sbMCg1CfFg8xxezBliM5WgE3f+3FUgU61MP4d1avYlwg77CkN6XWpDGJN4+mmoLBNknWlaD0eOo/FGlzEP39G+p0JYeSop1CzVRzzjz8kE/4JVJZOX1ZjvyoCdYix0mZFn6YESzNacn95xRpmvpoiMo848yTcRd8dqdzaoqaN3sCFm0g5nx937Whyhkgv8IykCOnbjFuD+L2VACnnm5tskcSCW95a746a1/0Tvpw99RvzPuBa2xxSRhSJ/igoG5TcgefTjTmPs05RRJi5qRdjI7AofLGZ4jPQWvJEWOyatlQ+0GkxR+5TX44J2CbmmSbyUGmQvmbovC9XE8BQNrP0+zsoCn1+ESEuE5iybCnfZRVwiJEZ3t4HRIyvmJR7CVR0WOP8527De69mpTzHvtDnC/Lghv5QfW+J80iUgiXAUHwFNaCJQejmjCCAkzwsH2za3vrfgib0p11DD7JHscYnl29vT/kjowB56p+BcpXEslfQZdyiQR0VUnmGeE9truGMwsbh5hZkRmWSQuFie3r5gtaxbhbHnTsaShd7cg/wLDM5jNI2aFjXHZgg73ltBxfucBpPPn/OJx/tpQX1t9pwYEVYEaujXd81IsHWEPNOpJRpWGau/eC3gKc5WqxNeBlE5fW8WAokoHe2cOPwu2IyWOLJi2IZz4OefTlu8ftzBn8FhvrfUKcVOXDAONug1bdET3Zqyp3x/qdLI+aO+iIe/WjDuwWoySuYPC7qLdcxzer3sW8AWe/ont3jm+CTrYuV8qoSppHZoQrqkCouKF1kcJXkhiK0Hlch78ZsAMkWqjFhBPp9 8zSc4qpz WDcI9saAu9MTvyZ6fnXbbPtiM2UnzTuGg1nS3XATVmeUv8++v8wq1jL/6Emz8ItySOS434HTDoMVW+VEpECOg3J6kIC9NbyL0MoFw+yOn0r87dC8YChP9ilzwqyb00PVHFWN88bjFHnTd1kU387H5108GfRqWenPDpL7z4rK9lGpsj8gto2SOfpPVAy6awkmbXUnEcOlBiCUmGZvXERi7HuNipwn67+OElCxvDQmxR5l0CrbJ2ABCEKOMtz1baUdgvxW9yPegsCsWhBRW7E5O4n6F1OsgimQToklnWpQBIOCxcSZQF+yUEuzjWyhET8kfE+MiTdK69obpYZ0iVzMYSFCS02bqBQ9cjxUU5vwv8sjlNAO3YoTw9A+095z04kAanuyYJ7QIUmnouq4i7vEa7k2qYgfVuhEXnwOg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett The walk to destroy the nodes was not always setting the node type and would result in a destroy method potentially using the values as nodes. Avoid this by setting the correct node types. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 73 ++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 62 insertions(+), 11 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index a748938ad2e9..a11eea943f8d 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -897,6 +897,44 @@ static inline void ma_set_meta(struct maple_node *mn, enum maple_type mt, meta->end = end; } +/* + * mas_clear_meta() - clear the metadata information of a node, if it exists + * @mas: The maple state + * @mn: The maple node + * @mt: The maple node type + * @offset: The offset of the highest sub-gap in this node. + * @end: The end of the data in this node. + */ +static inline void mas_clear_meta(struct ma_state *mas, struct maple_node *mn, + enum maple_type mt) +{ + struct maple_metadata *meta; + unsigned long *pivots; + void __rcu **slots; + void *next; + + switch (mt) { + case maple_range_64: + pivots = mn->mr64.pivot; + if (unlikely(pivots[MAPLE_RANGE64_SLOTS - 2])) { + slots = mn->mr64.slot; + next = mas_slot_locked(mas, slots, + MAPLE_RANGE64_SLOTS - 1); + if (unlikely((mte_to_node(next) && mte_node_type(next)))) + return; /* The last slot is a node, no metadata */ + } + fallthrough; + case maple_arange_64: + meta = ma_meta(mn, mt); + break; + default: + return; + } + + meta->gap = 0; + meta->end = 0; +} + /* * ma_meta_end() - Get the data end of a node from the metadata * @mn: The maple node @@ -5448,20 +5486,22 @@ static inline int mas_rev_alloc(struct ma_state *mas, unsigned long min, * mas_dead_leaves() - Mark all leaves of a node as dead. * @mas: The maple state * @slots: Pointer to the slot array + * @type: The maple node type * * Must hold the write lock. * * Return: The number of leaves marked as dead. */ static inline -unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) +unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, + enum maple_type mt) { struct maple_node *node; enum maple_type type; void *entry; int offset; - for (offset = 0; offset < mt_slot_count(mas->node); offset++) { + for (offset = 0; offset < mt_slots[mt]; offset++) { entry = mas_slot_locked(mas, slots, offset); type = mte_node_type(entry); node = mte_to_node(entry); @@ -5480,14 +5520,13 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots) static void __rcu **mas_dead_walk(struct ma_state *mas, unsigned char offset) { - struct maple_node *node, *next; + struct maple_node *next; void __rcu **slots = NULL; next = mas_mn(mas); do { - mas->node = ma_enode_ptr(next); - node = mas_mn(mas); - slots = ma_slots(node, node->type); + mas->node = mt_mk_node(next, next->type); + slots = ma_slots(next, next->type); next = mas_slot_locked(mas, slots, offset); offset = 0; } while (!ma_is_leaf(next->type)); @@ -5551,11 +5590,14 @@ static inline void __rcu **mas_destroy_descend(struct ma_state *mas, node = mas_mn(mas); slots = ma_slots(node, mte_node_type(mas->node)); next = mas_slot_locked(mas, slots, 0); - if ((mte_dead_node(next))) + if ((mte_dead_node(next))) { + mte_to_node(next)->type = mte_node_type(next); next = mas_slot_locked(mas, slots, 1); + } mte_set_node_dead(mas->node); node->type = mte_node_type(mas->node); + mas_clear_meta(mas, node, node->type); node->piv_parent = prev; node->parent_slot = offset; offset = 0; @@ -5575,13 +5617,18 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, MA_STATE(mas, &mt, 0, 0); - if (mte_is_leaf(enode)) + mas.node = enode; + if (mte_is_leaf(enode)) { + node->type = mte_node_type(enode); goto free_leaf; + } + ma_flags &= ~MT_FLAGS_LOCK_MASK; mt_init_flags(&mt, ma_flags); mas_lock(&mas); - mas.node = start = enode; + mte_to_node(enode)->ma_flags = ma_flags; + start = enode; slots = mas_destroy_descend(&mas, start, 0); node = mas_mn(&mas); do { @@ -5589,7 +5636,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, unsigned char offset; struct maple_enode *parent, *tmp; - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); offset = node->parent_slot + 1; @@ -5613,7 +5661,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, } while (start != mas.node); node = mas_mn(&mas); - node->slot_len = mas_dead_leaves(&mas, slots); + node->type = mte_node_type(mas.node); + node->slot_len = mas_dead_leaves(&mas, slots, node->type); if (free) mt_free_bulk(node->slot_len, slots); @@ -5623,6 +5672,8 @@ static void mt_destroy_walk(struct maple_enode *enode, unsigned char ma_flags, free_leaf: if (free) mt_free_rcu(&node->rcu); + else + mas_clear_meta(&mas, node, node->type); } /* From patchwork Mon Jan 9 20:52:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094270 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11AE5C61DB3 for ; Mon, 9 Jan 2023 20:53:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A34558E0008; Mon, 9 Jan 2023 15:53:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E5108E0001; Mon, 9 Jan 2023 15:53:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8373F8E0008; Mon, 9 Jan 2023 15:53:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 748DF8E0001 for ; Mon, 9 Jan 2023 15:53:53 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 479A6A0BEE for ; Mon, 9 Jan 2023 20:53:53 +0000 (UTC) X-FDA: 80336462346.15.5900231 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf18.hostedemail.com (Postfix) with ESMTP id A74FC1C000C for ; Mon, 9 Jan 2023 20:53:51 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fcpOTy1E; spf=pass (imf18.hostedemail.com: domain of 33n68YwYKCPElnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=33n68YwYKCPElnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297631; a=rsa-sha256; cv=none; b=SUr3XmJA1EkR8oSCMevxU9VNK/94GazSeOuiPokFLCQ7zo88FTh9x59KQwfHQ0khoNHgyo 1vzC3RqQZXu4wmaQTH4fUILMvG6nwPf9J8tPDnfkv/2RYEVs0PqGbr/DJJYh+QYEqYgQ/o lzWUNEvIPUYYpDkuQC/2SjUsNAhOdq0= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fcpOTy1E; spf=pass (imf18.hostedemail.com: domain of 33n68YwYKCPElnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=33n68YwYKCPElnkXgUZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297631; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xyMpIuQX3h8xEF8sgRu6qp18jXndG+tVBpDaAHqFUcE=; b=PUemKTeQZ5NUyk6IUzhDp5uj16LuzCzURIZ0TRpg94hCP2rT8ru1ao4IsSCb5d80OYgjwR qWILspwtPZsch8WjaW9yh0I3q5hh1X6RiBQwOSMDYbXSwd/fUwOYM4QnHswMEiN9SF/e8x a6gfFNaSOqHQ3hIdIoYyuivUfgT/h8c= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-4c6bd2981d8so66527627b3.2 for ; Mon, 09 Jan 2023 12:53:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xyMpIuQX3h8xEF8sgRu6qp18jXndG+tVBpDaAHqFUcE=; b=fcpOTy1E+3ThlAgpg3hHomgF7jysjdpHFVliFHHbaTXw696G+d4UafDfpkipyR4nuU tlOJBreetd3v3dCXyb/Rdewxh5p1khH4cqHcd7IJy4L2zy/b/nqItrk1BUqrfoTrE/cO 7u4Zqnf/OqJvpx/Forf/3GSWw4rGhAxROXbd0Dn4cjHGhWxv2F6/FkGrRQlT4BSVn9QB Gq9sxTxT4dzmvld2wEbl8PkQKZUyemG56T9+2h4MZ3hLbr2uHjNUXIbifPrLJAMZf98P qyO5BkbAlxHe07UMVtRr4iqChj93uPwUR0TOsMr/XaxeajQHkPY5F6oCUcd/VOMyVzz5 1yLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xyMpIuQX3h8xEF8sgRu6qp18jXndG+tVBpDaAHqFUcE=; b=AsB2ysbA8hT86UPEh+kyfgBRgiIKW5HXpwkx/rqk+LONYMdWYhlIWij2l184Up5ZdP hRxzn6ZAw3X1qcf2/AfW9hcU17vohe9n8kACN5n1PI8d8QUX8W6Kf9aVxuxTnYKpxJSO vWpXgr9tyXdy8tMfbtco5Ig+scMl3QfD0yPxf9bVgNyZ6W0wkbgDuxJ8lpncb1gDEkUZ lBifqdN4NCd2slfAU98MfFsgqyU7GUDmKtnKolExsqgBOdSTSU6etONy2hHgfrlY4SX3 w3y9/OkkjHUfJUEEdbLInLsNVHNrboCOPpIYKCnUJ8myhaaxPp0La0uKOzKMrif33JA5 dfyw== X-Gm-Message-State: AFqh2koOqk+py8GT0vApEkkhEPNCkk6lZmkwjR/Sio7CKqchbPJiHzjy 38LNAz1IkRNWPnf7RtKZ6lrXe0gddAI= X-Google-Smtp-Source: AMrXdXs29F0iyIf9MRCODAQbmPmP2d9pU1gWPBdnTE2PRXcNw1d8zc+w15TjL10LubrEOMnWhSTePMV2FM0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a05:690c:299:b0:370:2d8c:8193 with SMTP id bf25-20020a05690c029900b003702d8c8193mr1225043ywb.221.1673297630870; Mon, 09 Jan 2023 12:53:50 -0800 (PST) Date: Mon, 9 Jan 2023 12:52:59 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-5-surenb@google.com> Subject: [PATCH 04/41] maple_tree: remove extra smp_wmb() from mas_dead_leaves() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, Liam Howlett X-Rspam-User: X-Rspamd-Queue-Id: A74FC1C000C X-Rspamd-Server: rspam01 X-Stat-Signature: zeuw13jjkomim1wk51ue1afy5i9m4cqh X-HE-Tag: 1673297631-891401 X-HE-Meta: U2FsdGVkX1+rju9ilmumEJfwnOmPlzGlPDGkUKhLZn87aKKk12IT6XHS8S1a+oNpOBi1RaxEYK6kVz/pFSCv8tSDNe7G+ErlT0kplrHTTKxYQiPys1+qitRMpmeKhl1k1IOo/BCgHsCDftBki0/LFZtywPbuQDZoA+fEMPmo1OFjEGfmH14ZJBabL8qlAPBhs3aKvuyxbXLysNom7MuvYuQ+3av97gInC3ROjK+oSaQe7gIFcqfBQKy9ZfL3aHL1ynGOqL6jE/qOm4gYiyb3dtgbpU8R8cCz7VF62hRm9Z94bajqtUilJyPTyYbNNf/5vtBMAMLZgbV5cHZ09UpLEY9Gwz/qkpgOUyoBjCAlzAaaE8f461SU6URZADtjxvWTC6U+EUjRenWpSmHBbLO1qV5TZbmeScXUVPxCJUl97CXYF0M6tUbREyWpJeCpY5fCiSoFove4ZuWNj9NTGa0XLNHKda8GRudltdGN7Brmxz2weBzXvYCJiqTYP8UEqKzqwQeECfnfAlcg36+vkg/4rnJI5MqolwKtb0QIsgUG+axq8WRz5q4AW4sl2rASRhUhNFfXIrycY3JCJOw35hzg9CT0ePik6L2h6mRedI2ocZKE/ANNHtQgMYizDUVYyrHGv4QTwJDp5kug0JsHBZ5MCAf+E2GFkDLBmGV3bBh8Lkgiq6TruC/a3HNhAMCDYc5Ec/KTBft8TxMiy+5s0iksFDEDJ334I+Kh7TBji3yMF4vE0ob5LsSaUbNuMPQ92zy3QsXP61Wr36zdUepMNTlr6hEOf+cMOWUHQbMjUAxUwPu2MKzzS5tm3RB78IT71m2mKzLF6R8RvYX9eUbJGGiT/EHGAQAMYrLvHveg+Is8X1VdhvwXMSF6+1O1pyGHoacbJoytHBHtUzfRgkJs0c4KqQcMP4c8WrwbjnBVX+kYnpFqzPgKNwF2Ibky29oiDuORSyj861WZp2Ka1ybX7Lx Uhk+CII0 AE23PWdJhcFalNbPOGWD1g8ICtgizvvHP/YoVc+nHWlakKY1jLEPQKrMaMkZr/61wptxQLCuSvuKG9Vv0LN1zmXAbq8z1uz/M5369DVc+FPRjEKfcE06/zWq0DJmr58B+WpeDLjh4dHkwchIixXdqBScpbe8JFmRHQifLuW0IeSVZ6PdYEXGvrS7nKce2zZD367jOOBxjgO6/c2XGzCMl1YDNBKrLil4a9fLNJUeMhq72rGs6UBrtDP37V6DnCpY8g9YVR9+tCcymc4r2i2ONd0hyfwT/Is0d2/guzoRPYUULkumVISLRZqLDxYSC2bdjvj4DN/IVm/Op7ZeTZbzDkn3flwVyOzN6X1jRSW+82nnQroZVgjZwOHT9oKTUg0KdDDH7O9ViXjKdoBvjIbAvAWZ6dnhyArq91sPS X-Bogosity: Ham, tests=bogofilter, spamicity=0.002250, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Liam Howlett The call to mte_set_dead_node() before the smp_wmb() already calls smp_wmb() so this is not needed. This is an optimization for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index a11eea943f8d..d85291b19f86 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -5510,7 +5510,6 @@ unsigned char mas_dead_leaves(struct ma_state *mas, void __rcu **slots, break; mte_set_node_dead(entry); - smp_wmb(); /* Needed for RCU */ node->type = type; rcu_assign_pointer(slots[offset], node); } From patchwork Mon Jan 9 20:53:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094271 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6B52C61DB3 for ; Mon, 9 Jan 2023 20:53:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B6268E0009; Mon, 9 Jan 2023 15:53:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 466948E0001; Mon, 9 Jan 2023 15:53:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3076E8E0009; Mon, 9 Jan 2023 15:53:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 20F8D8E0001 for ; Mon, 9 Jan 2023 15:53:56 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F05F8AB11A for ; Mon, 9 Jan 2023 20:53:55 +0000 (UTC) X-FDA: 80336462430.28.7F3E2AC Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 57B814000D for ; Mon, 9 Jan 2023 20:53:54 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XZhB0Mur; spf=pass (imf04.hostedemail.com: domain of 34X68YwYKCPQoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34X68YwYKCPQoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297634; a=rsa-sha256; cv=none; b=KkGTDcq9BKZnBBnYCSEKrzg4tMBkjWHAJOahg0VtQl9y7T28bhheeY42ICQSx4hGwzeB5X 21iUgWX96r1dr7AOKWLyjicndpMMsLnwocDehaGUzZdseI/uYuvbScTb0eBqDYvAcdK5DV sNNeqaIfiVoGNgEgSa3sKVyKgIkD+9Y= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=XZhB0Mur; spf=pass (imf04.hostedemail.com: domain of 34X68YwYKCPQoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=34X68YwYKCPQoqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297634; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r/ByKXL5/TL05Cni5RZ8/3QBk5f4BKrzjgLm3Nsczuc=; b=sr/bj4SbmzCiQZzRFNS5OmV6dhPTA2OM9WZLRiS0IgM4olViueV4Z1a5s067UR7Dmcs8BW izLoluCCf8Cj0KmNomqBTsG5mtr5vCvn0zi+HCLfTw0ypvX7cmkBFCyh4sbtHmmx10pQDa vBWL0XiXx1T1WYBBuL7hUJK9JlaK6s8= Received: by mail-yb1-f201.google.com with SMTP id 81-20020a250b54000000b007c002e178dfso1324296ybl.9 for ; Mon, 09 Jan 2023 12:53:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=r/ByKXL5/TL05Cni5RZ8/3QBk5f4BKrzjgLm3Nsczuc=; b=XZhB0MurOIdgvWrQwu+hJ2lSxOQT+CHAxznZAlATY9tL6yBOIdJSyl09j7qs++7K6+ +noh5D5/c9QmPEOhqfvJHyQwq3kACUyssJheSfMwcYViJIFHS1/y1fiRpO9p3lukpSo8 zurDxJt/kmbPIHQRSNmtoT5iYyxem6gKfY8wpBL8JKkSoFXDKg0JciA3ABN5+4Pa/8FZ Svy0a58Amowv53Vk76UKApMAQ9ZprfLfSHlZL0fWzl8fmhDrZgAmkrOVH9/RFI/nNlnR WsAxkAJv9sUdL+8FTIMmDBza+16v8vXL8ENtWQu+olGHSLyyl/6A++OsnTnGdWDPMYeD K8RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=r/ByKXL5/TL05Cni5RZ8/3QBk5f4BKrzjgLm3Nsczuc=; b=k1ipAbJIIrEmeKqsr3ZKJs6BXjKDrAQIaqSGUHExMXINJCNyqGJFqKGoMLezlin+fk Hfvamw9/UK0kZ9tkht0y+Ic/lkL1Iarnfnp4m3+8TeMmIXdOCQMXwj0KbD8a3EkSQfVJ DkzEASxEigw3nnSKH55A7zll3OtDjt6KneKMJMWovJDZeJ2JwPgnOQQmVX2JzfiBvF2U MEvMkh2ZEUCaTGnDYO3S1eKIj5EfLN3RR6BPEs9492Se/T+z5SzlkazLXkONK8ubXArx tTjWMrhTHeOW9pIjrHKO28icefu0suXDmf6/pRGShAow733XVuqMBB57ysAo1L7eRRQl UB5A== X-Gm-Message-State: AFqh2krfY7IwYdsWhzPv3taZC2Y+FFr7fKCkTfHq/i0pdYkHoL0zVWbJ Pt4EL0qeeO2bTd/Cc5y+X6GycE53K6s= X-Google-Smtp-Source: AMrXdXv6pxAJ0Ez+UVi7qfPaeqrQmlVnO2LvHLtbQu4X0J/l7RkUZy3CJtaYn7PIXuCjX2fZCm6e/eZrYK0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:f007:0:b0:388:941:23a8 with SMTP id z7-20020a0df007000000b00388094123a8mr1505536ywe.152.1673297633444; Mon, 09 Jan 2023 12:53:53 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:00 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-6-surenb@google.com> Subject: [PATCH 05/41] maple_tree: Fix write memory barrier of nodes once dead for RCU mode From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Rspamd-Queue-Id: 57B814000D X-Rspamd-Server: rspam01 X-Stat-Signature: pq4f6a1y4i91j3h6ecop1wqchonfmjb8 X-HE-Tag: 1673297634-420735 X-HE-Meta: U2FsdGVkX1/3zekSpojF3H7e8rWRvMWjKmsJQ5xtJavgHIfxh7UYtCIHzCRufJaZTi2tBADgVC3s5B44gotUDZSf0QKwHrFcZq+D3kR7EzfD5MtsHshk8wdjed6iKx5YQFMh5Y8rTQTKegFiNDwbio9j32UfzhYKpMxGx6WoU3/nPlLOJ6///R65uDPEvZ0AtreaVUbBOIwodDPg5khWKR2TcG1F3CKFFYQ5d9NnNcTtW/WDDIDwtl1uY20dLCuapGBR2ztL7BZB5MXVc412l7AUf8+vEvoHkUxHOvVQMe/xTfgMdPNq1ILOjbzTzFIKrtBBXO2kqvH+TsJ+oMlQRm94GOVcZuOEuY6+YVxhVuA7d7/yGPYfZq7irQcbiVHOX0jaNo6LyzctmrLpY/KhJ5dfXKlIuQrpGneclcPjt4dhjTrOuqP2FubIgFCeNVjO/a0AONBiKo6Uy8sbcX6PH7sftewlTvkPTnXIkb6o3VIPi7FU1k+eWcyx8jsEBd+KXIUIzNqAA+VwG0B1/rU8lKeF6VbIYx0wRJ4lL996RJLA7xLVtCDIYJm9yQj57vSkf4kmzDQSLHWCBzSD7EyyMFC0bbruSzp28gFqPu8qvh113ggsHquks9t/OXc661lYbwIAbgODmfKjiQz+jalaIAoR+C0+BIJ7V+8wncAFHrsKbQPf6CGGupvmHY2QO8KaIuwYTBmQRcbO7fxv0ozGoMqXU7Ay650gzF/uP1ImAOkW2rdwnyWgdjHCVWNxx427/Oio2qBqUzorUxmEHHyJaFvm7GzlXbszNA0nRQ6+qI+/Gs6p0JHtoY2AMcp0K1CD75ouFKPad9tq6atXy0CQZkW2AP3vRE+bsmJ7yHl6rGAip98JGuwAd9vQvc4uVWBND1nWBhbtG6RiNv4kqdWK4u/Yx93tXYEuAWd+0CjGT+EN4P+JdNV0GvfdV3K2Vim6tLajk0nsXVoJ30lIiBp 9wEFHD8d TP/DRC0pM2gK8Jpvql59yAAzVY/YVj9gH6M4YAGHpyTNLaKlCrPMXeHFAoAINvSNWgPKcH6IUgeUy+kfV2wAsxV/vA9lc+L10677WmPJdgeXACNtMndJr9x/gShcI6wKBY6wfUIIMrYb1hTb7uMVmhr8LU4wxME+yTvky9t3L6gy/F0jnx2/MShETXLVjuElAFJ9rH/AcJEx5XikBeKCzHRA/xfX1e78TFrK9hzBMfKxpnFjx4qA9K+UFl9UO1Pja9bzehIkKfbasgbkF9yjTV0XLOHOTgpBGTXoIW0OruOUsY3puhxUOTDkOKxr4WaeDchohiHnyW5DYqkFYL3gksUYYfIEXLh3y6LljNwUKeKeOYnPif7UzWHQHR0Xcwi183ouH7KKYGnFIQC/HAJvaBr+cTPs4XSlx2Hix X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" During the development of the maple tree, the strategy of freeing multiple nodes changed and, in the process, the pivots were reused to store pointers to dead nodes. To ensure the readers see accurate pivots, the writers need to mark the nodes as dead and call smp_wmb() to ensure any readers can identify the node as dead before using the pivot values. There were two places where the old method of marking the node as dead without smp_wmb() were being used, which resulted in RCU readers seeing the wrong pivot value before seeing the node was dead. Fix this race condition by using mte_set_node_dead() which has the smp_wmb() call to ensure the race is closed. Add a WARN_ON() to the ma_free_rcu() call to ensure all nodes being freed are marked as dead to ensure there are no other call paths besides the two updated paths. This is necessary for the RCU mode of the maple tree. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 7 +++++-- tools/testing/radix-tree/maple.c | 16 ++++++++++++++++ 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index d85291b19f86..8066fb1e8ec9 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -179,7 +179,7 @@ static void mt_free_rcu(struct rcu_head *head) */ static void ma_free_rcu(struct maple_node *node) { - node->parent = ma_parent_ptr(node); + WARN_ON(node->parent != ma_parent_ptr(node)); call_rcu(&node->rcu, mt_free_rcu); } @@ -1775,8 +1775,10 @@ static inline void mas_replace(struct ma_state *mas, bool advanced) rcu_assign_pointer(slots[offset], mas->node); } - if (!advanced) + if (!advanced) { + mte_set_node_dead(old_enode); mas_free(mas, old_enode); + } } /* @@ -4217,6 +4219,7 @@ static inline bool mas_wr_node_store(struct ma_wr_state *wr_mas) done: mas_leaf_set_meta(mas, newnode, dst_pivots, maple_leaf_64, new_end); if (in_rcu) { + mte_set_node_dead(mas->node); mas->node = mt_mk_node(newnode, wr_mas->type); mas_replace(mas, false); } else { diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c index 81fa7ec2e66a..2539ad6c4777 100644 --- a/tools/testing/radix-tree/maple.c +++ b/tools/testing/radix-tree/maple.c @@ -108,6 +108,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mn->slot[1] != NULL); MT_BUG_ON(mt, mas_allocated(&mas) != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas.node = MAS_START; mas_nomem(&mas, GFP_KERNEL); @@ -160,6 +161,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i); MT_BUG_ON(mt, !mn); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -192,6 +194,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != i - 1); MT_BUG_ON(mt, !mn); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } @@ -210,6 +213,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); MT_BUG_ON(mt, mas_allocated(&mas) != j - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -233,6 +237,7 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != i - j); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1); } @@ -269,6 +274,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -294,6 +300,7 @@ static noinline void check_new_node(struct maple_tree *mt) mn = mas_pop_node(&mas2); /* get the next node. */ MT_BUG_ON(mt, mn == NULL); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas2) != 0); @@ -334,10 +341,12 @@ static noinline void check_new_node(struct maple_tree *mt) MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2); mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) { mn = mas_pop_node(&mas); MT_BUG_ON(mt, not_empty(mn)); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); } MT_BUG_ON(mt, mas_allocated(&mas) != 0); @@ -375,6 +384,7 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); @@ -382,10 +392,13 @@ static noinline void check_new_node(struct maple_tree *mt) mas_node_count(&mas, i); /* Request */ mas_nomem(&mas, GFP_KERNEL); /* Fill request */ mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mn = mas_pop_node(&mas); /* get the next node. */ + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); mas_destroy(&mas); } @@ -35369,6 +35382,7 @@ static noinline void check_prealloc(struct maple_tree *mt) MT_BUG_ON(mt, allocated != 1 + height * 3); mn = mas_pop_node(&mas); MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0); mas_destroy(&mas); @@ -35386,6 +35400,7 @@ static noinline void check_prealloc(struct maple_tree *mt) mas_destroy(&mas); allocated = mas_allocated(&mas); MT_BUG_ON(mt, allocated != 0); + mn->parent = ma_parent_ptr(mn); ma_free_rcu(mn); MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0); @@ -35756,6 +35771,7 @@ void farmer_tests(void) tree.ma_root = mt_mk_node(node, maple_leaf_64); mt_dump(&tree); + node->parent = ma_parent_ptr(node); ma_free_rcu(node); /* Check things that will make lockdep angry */ From patchwork Mon Jan 9 20:53:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094272 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D7D7C54EBD for ; Mon, 9 Jan 2023 20:53:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC8118E000A; Mon, 9 Jan 2023 15:53:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E79858E0001; Mon, 9 Jan 2023 15:53:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D19A38E000A; Mon, 9 Jan 2023 15:53:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C30AC8E0001 for ; Mon, 9 Jan 2023 15:53:58 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 91B991C65A2 for ; Mon, 9 Jan 2023 20:53:58 +0000 (UTC) X-FDA: 80336462556.14.4DEFB7B Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf02.hostedemail.com (Postfix) with ESMTP id ECAD18000A for ; Mon, 9 Jan 2023 20:53:56 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FGVUXzbM; spf=pass (imf02.hostedemail.com: domain of 35H68YwYKCPcrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=35H68YwYKCPcrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297637; a=rsa-sha256; cv=none; b=D3LAfTm9hBSBSLmrKOJHc7SxPuO9n/b/Nv2sxg3AhfJqlceHYTY0OimLaKWKwHaTEh+Hz5 Q64sW2jkUmJ7yVpRRS3R201V2y/NT+UZEpSX8aCm8m5QjlVJXAuDSvdAwRi3a9R6p+o4d0 UOX+ktToZW3BRSpCdgRNFEhVcR/F30E= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=FGVUXzbM; spf=pass (imf02.hostedemail.com: domain of 35H68YwYKCPcrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=35H68YwYKCPcrtqdmafnnfkd.bnlkhmtw-lljuZbj.nqf@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297637; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tpY24U65dxDevik5396IsQMXmVZXiLzmW2j6b7ptC2o=; b=8BDnCJ9FqCpwlsdPKVuprFDAVAbKMkUkVWgtnfwKfPSbjsYeuFC6ZRpIsfHChtlUfLcSdA HkXQqLiNypgsFsHYnLUNpzdx67T2yDtOC0TH/M8c8PCcY4fBAgS5XSFJ4RWsPiL+JVB5+f /QISsxaIyLKVwb7u+DdkSJyjsfs1JN0= Received: by mail-yb1-f201.google.com with SMTP id g9-20020a25bdc9000000b0073727a20239so10199864ybk.4 for ; Mon, 09 Jan 2023 12:53:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=tpY24U65dxDevik5396IsQMXmVZXiLzmW2j6b7ptC2o=; b=FGVUXzbMKsAgHtVe2H+ZGPByy9vTxmIYvrPV7ZuF76ZDV4PCOupJdRY2uHLiI7i+8R 4Fjm3rnOC/ry4DX8g5MN/0YQ4aHPdCQZYFwG7UcSZ/43mvYNjuR6DmjlzCEnl8acF7KD ERVaFO4iv0zgtEZ4MmYzhpcYCiFLTkatnznfDTOltPoYyw5CvEIb1yptgTPwDbjWQDix pGxXZ1tFpWIKQUAKj6j+gMffNX2JeuA3u7s8cWrwnHHzy9T6apLvHv6bApzCw3vZvb6B a2xiK595oO97JhT8ZQYkTmP/j11qoIyiGxI24X5jHCYtpJjinnDlcXs5UPFQKhdZ/Kdz DK0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=tpY24U65dxDevik5396IsQMXmVZXiLzmW2j6b7ptC2o=; b=qzdBiQsZYembzc/PrLvhk7ZVRjLawfbPj/5GcoUnaeOn9CY/SiOjqZT8s1vnsI9DIS N0Dzz0ERmGujYpiJJHUL2XbbVhQKhLB7B5qAYt2FJ2iPdgaPJepu+IgYaVdGmhVcEKPz i/xY20ww/G6FCrVbWV3kjhVXQy5tYYw4/rFXzp6uO8MlQaXiaHFMfKsdhS2vvrHgzHqo u3V4yOQqfswOyPLMTxZdQMywXLlXPqIv8nERkD7A5yoUuvm2HKUDHBJEQ3W7i3nfV/I/ YC+HiX4S07CzW3vNj7e9DXqiuBKmWeKdEA9WIqmO4LvlvP1V4XRjMBRJM1Ba70lUzb7r zNLw== X-Gm-Message-State: AFqh2koLqMWFUxR1gpYvRA6GI4lmkBqiOZdwLxAod3lDa+/x5yS34ocZ dBfRO8NwSVWVT0PiC9GeY+f+o/GRWoU= X-Google-Smtp-Source: AMrXdXs5jnkdpf0WLGjSqHJja91Z23ecX2Os+zrAvpCFIdrfHWytTqhc2UPXUEbTUa59RZZK2SYuIAkg8gM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:2491:0:b0:7ba:cd98:5bed with SMTP id k139-20020a252491000000b007bacd985bedmr646916ybk.69.1673297636065; Mon, 09 Jan 2023 12:53:56 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:01 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-7-surenb@google.com> Subject: [PATCH 06/41] maple_tree: Add smp_rmb() to dead node detection From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: ECAD18000A X-Stat-Signature: 4tmgi76i1dru6qa3naeoexecqusk91wg X-HE-Tag: 1673297636-826545 X-HE-Meta: U2FsdGVkX1/+Gfby7Hp+VCiPspZRpIEz1FzgtDEjZ6C+dqu8LNXG2ceoUZB4RLsuYAj5dbRKQbXjpWxrwHyEuQnSXc8flw5I2baaUZ2Bc0Wa4Thkkn9zJz3R8m7svtsr6i0y6+zdQeTE/TCU9X88e3n6dBgQ65JAiMmnD0ROzhxborq6ayj9h/LttyR4W6BixX7NYRR0g77HVdnyhbEhJ2EnF25inlXisIvBzp3JeTp7uLEMGvHrG7XDdIT28RY3cHpEAcvJH8ZTJ1mTAp/9acsfV09FY1x+eczMyy2iZasuV1Kyxqb/lSENJPp3v2WeEC4DRamlFLltcxZ90eeiKF+NXUuR7a3U8FFRa5UIMENWRqhNhr8pvkcybJNJ9aqHEZzO6dLhF/uzXslMz36U7yj/sF4l1IMfenFkBZTvqyXC2/Lm3/J4agzbm0mVdR8cdqv1gtHw2UNxieteA2I1K1nTAwxc+1qV3r3tF7eVrSb9DmoBZJ9G6JyIGpFAfskfCnvC8nISkYiUofqCPkd1zO2L/JP80nj5efxlP6HsE4toZTlEZ4u2RdcdlOnwle4/P6CVpto+5zAO6NRQuF+lVcbdGk1qB1KOLsWciR94Yof2Nv1FC/XEBtiM4LxL9ld4bklLUjhgqueXZyN32vsLW99R0bXzgNDquC/farUVNErl0591TBAutjAxSA//HhlN3NZdaZA3siLuKt87jEI+nA6gsf915VJDBjbBLF57RJVSs7FY9yJqFwMJUaQQivqqnFY3wnuHAcP5KynVHGS4XqbAZwowdyHohEPT7r9UriguMa5QZT7FAIGQGdzRy1tNi+zvUihkfKIHMHTcY85VJ6fLE/r7zivviqmjuuYj9rpOKsNuPbufiZpzVUYkYedBy7DWLM7YbLZ//jm+IkRR7hyRfVxcDKjdwPfEM4ymtbZUrmHh2rVghVr8kV+d/S5eQzhjHezZnCFVGGBQ7oh 1Mt/FcDn mnLVkARotW3Vrn9lwBBq4LfJBXk82j+PatipawFeslbde1odd22qqyMQNCaVBhqO+Np8uXB/+nWjXs/b+SQYzpi1jSJfjnbJm1oA7zhXlqt+GAW+jU/XTZhRQ0xpQVHjGKqJR8Y7+py04jbQB4VvEvWE+SkuY1se6OwGAil558hKFu8rqyrubX0G6VE+uwjn3TMjXU+/MT6PqK3Ltnw/TFV421zqbWeo1MpkqGFY3shDooQjI+/ZBAw1hYq8r5vsJj43gI55LLjeaBaCPihHQyfHWUJNCWQS2LThE5agFm1pZT1S3pYtuTaLRdMPbv7Ll8rbzsTbkevRAi3E0Y2ZjUvvhw6l2UcUi/PGW7tX0jdCTkJ7DDRKvsv0nze/n6SrN6JtG+jS7zxGlaIvVAwWdXOu71Ipmslwx3ayv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Add an smp_rmb() before reading the parent pointer to ensure that anything read from the node prior to the parent pointer hasn't been reordered ahead of this check. The is necessary for RCU mode. Fixes: 54a611b60590 ("Maple Tree: add new data structure") Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- lib/maple_tree.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/maple_tree.c b/lib/maple_tree.c index 8066fb1e8ec9..80ca28b656d3 100644 --- a/lib/maple_tree.c +++ b/lib/maple_tree.c @@ -535,9 +535,11 @@ static inline struct maple_node *mte_parent(const struct maple_enode *enode) */ static inline bool ma_dead_node(const struct maple_node *node) { - struct maple_node *parent = (void *)((unsigned long) - node->parent & ~MAPLE_NODE_MASK); + struct maple_node *parent; + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); + parent = (void *)((unsigned long) node->parent & ~MAPLE_NODE_MASK); return (parent == node); } @@ -552,6 +554,8 @@ static inline bool mte_dead_node(const struct maple_enode *enode) struct maple_node *parent, *node; node = mte_to_node(enode); + /* Do not reorder reads from the node prior to the parent check */ + smp_rmb(); parent = mte_parent(enode); return (parent == node); } From patchwork Mon Jan 9 20:53:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC237C61DB3 for ; Mon, 9 Jan 2023 20:54:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 469508E000B; Mon, 9 Jan 2023 15:54:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F0FF8E0001; Mon, 9 Jan 2023 15:54:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26B528E000B; Mon, 9 Jan 2023 15:54:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 184298E0001 for ; Mon, 9 Jan 2023 15:54:01 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E6720120BDE for ; Mon, 9 Jan 2023 20:54:00 +0000 (UTC) X-FDA: 80336462640.30.72274ED Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf07.hostedemail.com (Postfix) with ESMTP id 54B9340013 for ; Mon, 9 Jan 2023 20:53:59 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Czdnd0b+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 35n68YwYKCPktvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=35n68YwYKCPktvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297639; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TT+2XsH00KwTX585kTrU5AriHJozMAXcs0iFxeAd0aM=; b=swpmpASH/gc/nBQLez9nwJWs26WLngF44GMYGevL6Iohbk6BPxh1G/mS4Iz4p3drBPIKiL Q2TP0JAwZe36rmDV+GjFjDiW9FXAbg8gQKGQb64RSZKfCVIRdsuK1t5Wzk2sdwlpgUgP8T Vdb7wA+BEV6Bu2wmWPmAiRHOch8icM0= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Czdnd0b+; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of 35n68YwYKCPktvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=35n68YwYKCPktvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297639; a=rsa-sha256; cv=none; b=lALwaEc4ejDx/NbFT/ZYVnhNldU70bra0SZ4zlEWhFGsi+vW129BbnzWtLa15HBqrT0h50 msj0YtZNfTD9XNvzDNBV7xH0cOMLXpFAE7nrP9JEGNzMZABKpwSmIrEIA2xOMT/HyXYKo/ SEgvAOYdF+0/TGzNKbBvghtZQ1hMdFo= Received: by mail-yb1-f201.google.com with SMTP id h66-20020a252145000000b0071a7340eea9so10190379ybh.6 for ; Mon, 09 Jan 2023 12:53:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TT+2XsH00KwTX585kTrU5AriHJozMAXcs0iFxeAd0aM=; b=Czdnd0b+5bkyGxVk+9aViJOW1TJ5m6k5XpyXdj4R+Q3dGGEBK2eP5deQvySgSh0iz/ GZ1iYooFrZ5CipGLrqMEQsrY/Zn4kUyQA8m4xIFN7mfEerPKCDkU1AubZLdIfDfth4hu d8dG+6QDcTUs7A9Y052NxeL9TA5a3QdfxlEt4CBoSlbUsLAUBEORkFyS1sdo7qsKHieD EXHjCZRBRMhYEWoMQiXcySJyrWxVf33iALFZt6xsCbFcNo15dZaOnJt+J48Z17ETAj7O 45DKwd6DY6kCK0yPohqOb3E3TN0P5VYKNR4qSRkuKInUqGltfhkGbi1MmOGm2N8pUB+H +TEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TT+2XsH00KwTX585kTrU5AriHJozMAXcs0iFxeAd0aM=; b=tYQBLzigA3kp2lx9oqCo8dgtFFgoxM2IYgs28z7F4zypaoYnWxPGaYYtBdhTdawhYM Sie19nsQJf9BfgMaZlE3YUvo1HsX18aZcCIQTxKAGvrDcxhKXDutFzPBYhQIrTMU+dZl 6uH8ZxU6rxtCaW/mi1IJiVmpUonqM4QlW4PDqfCGUCL0UbhZUmp3T16yt0oOMjdUizQd Y15iMrP/nO0bAQwthQuxlBV3sCwIIGhoFtDnxEvy+7KQeGksUyWN1Dtm3p0GAIVp4xsw Wxp/NqamQJiiZDB+fRRGNC8rbX7nNIYI9mivWl8zbuTtm2F3rN5A1Q4NBDcWTTmyD425 4MEw== X-Gm-Message-State: AFqh2kpNpdqe8GXfVvB8haoiaTaXwGTteJ5yzGvoxwV4L3tcUQV8TUSy RNyjgvesX4O7ltr4pIWpAiOCk6qns8k= X-Google-Smtp-Source: AMrXdXtUN3SiDpbjRFPvhoNJBXB5uM1KkpI6Zcz0Naqt9m0yZDl1B9u4vz5wIeviZulbrZU/bcJYjep5MGE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:abea:0:b0:762:b86:e82e with SMTP id v97-20020a25abea000000b007620b86e82emr7288109ybi.407.1673297638462; Mon, 09 Jan 2023 12:53:58 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:02 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-8-surenb@google.com> Subject: [PATCH 07/41] mm: Enable maple tree RCU mode by default. From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com, "Liam R. Howlett" X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 54B9340013 X-Stat-Signature: efdcxmqiyt7huu5zps9i41s6r1k6geaa X-HE-Tag: 1673297639-665336 X-HE-Meta: U2FsdGVkX19p3a2VPbbahEi/i9+EEX6WZ7kRJpNwznD14XachIaXRKG2m0hJcxm+Nifrt4yYSqFYrKFy0Wm0zxdpvsUQbOCIg6wFPh3RaObrDDJy1qjbdZjXCnyuzsL+67GDMSwwvjK70ckAVyyQz4ExgW1zyfItCUMUMBtsy9+07AQNWaNcXpRMSDD1LABWie/Ant8CKC4dudiCEgTNdBgbIBQsNIL4iDmxGHZJOdYGd4ZteUy4aN67fKgfTsD9be/PdceFvJpipkUfVpevfU4NLieZBo1c9bkcCRh6pdcrdH+wZziNu+/9Y5E6d/MKuYFKj3l7RZY2YbvGYWRROUftkjlrCKdEX4dXw6iUWsl5780Mm32xm1IrvI3cPA7o3Atvg0Eb8USMAPDHCPMgmLoZJUO1XcKKRpVjBMQWjK9x2J7ycvQwDTqVZkOzC3crhCYIXsGvrouyn8y7nDG0Ymhqys5foDoQrMTn4ouxZwXN74IS+XiYPWyhM1wdUiAE49wDcm1pIuVcQlcspWo4q58UilUYbXgb8r4KKBWANRQf46f5oeYgH56D5rpbpQoo1Bt0FUJBQa9/G92f79TzGC0IxYkKHAtb9V8dn382QnFIjxovZ9NrNKjywsKLzOs0UNNpUjXj7ixDrNqq8jVZ1Q+egVBZoak5qSzdpDUnS0yK1ZFucvulenhK76B25CllYDahN4Eoyre55PAyZuWRSib/YZDtZ9viiIdqNJZYfguILqpIZ00XqWkidkncd2QUgRcP+e4E20Je9IofLj24y6jKIKKqC+laUyv6yIA+nhDTNiy2JksUmpNrx6xVDwv9n5E5CnkTKB0O9bF8W4UjYHXbHLaxFN4tSRfWvEwtk7SyaEKzfSgOy+3GiMd7mXI1kqoHODkVnDUp8/4yOjugLZkWRZlXpkzIDphgYnWYQrFeVyalmAAwWXiOPx+yJBzjzWnFK1dsORHrIF1TdM4 EPxCOzrm sp9B73Dwzc69EVbvpWPSaMoidgJtJTZ6vSoD6QCt0VuZvYwxUPLeU1V/OB+uJCwCHGtR2agRXn6/F3DACQMnN5O/Dvx707KRPZRyrSNYYYk1gpjep4q5Bl4MaS8J8U0WV3VYIHyt3GgNsllyPN0uUtOoirAVeDL5lJncOYkvS+Hi7isBDjxur3Kx5gWYTEqWhNP6ugB2y2pmMr1eRAQXDMUIY8mIKe8eLht9INZxMGyKMRMg9mmOFhs2zbg1XzPvVLP3FzZMyViPIxpPq6ub9+CsgA00mKa7oQic+XwTv10A6Po86x22nh1vhX5e/kEb1cyJuTYG9HFnlzjDfJqmYMfKhMVERypT7w5NjHIGZANw00/I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Liam R. Howlett" Use the maple tree in RCU mode for VMA tracking. This is necessary for the use of per-VMA locking. RCU mode is enabled by default but disabled when exiting an mm and for the new tree during a fork. Also enable RCU for the tree used in munmap operations to ensure the nodes remain valid for readers. Signed-off-by: Liam R. Howlett Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 3 ++- kernel/fork.c | 3 +++ mm/mmap.c | 4 +++- 3 files changed, 8 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3b8475007734..4b6bce73fbb4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -810,7 +810,8 @@ struct mm_struct { unsigned long cpu_bitmap[]; }; -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN) +#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ + MT_FLAGS_USE_RCU) extern struct mm_struct init_mm; /* Pointer magic because the dynamic array size confuses some compilers. */ diff --git a/kernel/fork.c b/kernel/fork.c index 9f7fe3541897..58aab6c889a4 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -617,6 +617,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, if (retval) goto out; + mt_clear_in_rcu(mas.tree); mas_for_each(&old_mas, mpnt, ULONG_MAX) { struct file *file; @@ -703,6 +704,8 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, retval = arch_dup_mmap(oldmm, mm); loop_out: mas_destroy(&mas); + if (!retval) + mt_set_in_rcu(mas.tree); out: mmap_write_unlock(mm); flush_tlb_mm(oldmm); diff --git a/mm/mmap.c b/mm/mmap.c index 87d929316d57..9db37adfc00a 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2304,7 +2304,8 @@ do_mas_align_munmap(struct ma_state *mas, struct vm_area_struct *vma, int count = 0; int error = -ENOMEM; MA_STATE(mas_detach, &mt_detach, 0, 0); - mt_init_flags(&mt_detach, MT_FLAGS_LOCK_EXTERN); + mt_init_flags(&mt_detach, mas->tree->ma_flags & + (MT_FLAGS_LOCK_MASK | MT_FLAGS_USE_RCU)); mt_set_external_lock(&mt_detach, &mm->mmap_lock); if (mas_preallocate(mas, vma, GFP_KERNEL)) @@ -3091,6 +3092,7 @@ void exit_mmap(struct mm_struct *mm) */ set_bit(MMF_OOM_SKIP, &mm->flags); mmap_write_lock(mm); + mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, USER_PGTABLES_CEILING); tlb_finish_mmu(&tlb); From patchwork Mon Jan 9 20:53:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094274 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53C28C54EBD for ; Mon, 9 Jan 2023 20:54:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E53678E000C; Mon, 9 Jan 2023 15:54:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E032C8E0001; Mon, 9 Jan 2023 15:54:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C86AC8E000C; Mon, 9 Jan 2023 15:54:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B965A8E0001 for ; Mon, 9 Jan 2023 15:54:03 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 746D280F31 for ; Mon, 9 Jan 2023 20:54:03 +0000 (UTC) X-FDA: 80336462766.16.BACFE0A Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf13.hostedemail.com (Postfix) with ESMTP id E13FE20015 for ; Mon, 9 Jan 2023 20:54:01 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hJQlgWQO; spf=pass (imf13.hostedemail.com: domain of 36H68YwYKCPsvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36H68YwYKCPsvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297641; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bU2TYXI/IQ+ahWPrvtMjvIvPP35LcWsiAJFQ60YWV8o=; b=muLchG3kYaI5wIjA6nftkun6WgvW9OTu/Z9u3ngC/SeEroXOMtTFug8Uxj5/UMe50IXjx/ 3Nkt41+AKm79hktPyMEpItAr8wSoYJaqQibpzgab1j45OYCNN3//rFvBAfv3SMGtQP1Ms3 zRwKMA4jER3WCjViqHQLBrFxwKNby68= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=hJQlgWQO; spf=pass (imf13.hostedemail.com: domain of 36H68YwYKCPsvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=36H68YwYKCPsvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297641; a=rsa-sha256; cv=none; b=TsE0nk8yDkxBjXeENM3EKUIf1J/U16jNhQG7yIdp0nrEmm6/nESxJcZNBIkeXZ/V3Zjew9 R0IXnlSGuSYwZ23ng52Or4LyEYwUdUxF0k0U2JTjTMblU0j2wV6Nw3tqWkq6ttpnG7AgVJ NixisormLK97WO9ROSeiBpHEhZZwQQQ= Received: by mail-yb1-f201.google.com with SMTP id k204-20020a256fd5000000b007b8b040bc50so8772972ybc.1 for ; Mon, 09 Jan 2023 12:54:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=bU2TYXI/IQ+ahWPrvtMjvIvPP35LcWsiAJFQ60YWV8o=; b=hJQlgWQOwjt9bdNS4iNCFQHGlLFypCJo9NibsA7lH+Cj66pCbZCU+CxVh9F7A2fm7T vmlH3bAYg1kSOc6vTNCehxfbrsiSwMkUwfklPcXaZBTBOgsQNuQ1lURfneHqEpASj7ak 72wsJeHj+71ppFuT3bry8xcf97dMVhsy87IzDxduellVAKg7BiJc4vIK8HA7ompTnbjq +gURuiw11LvdH0ad+L6DkctzinvpOQbzYQIPe9DQz0Et4Siymb/UJSUp4isr8x/TUfSb z+/96aen7RzYU/fA94YBNLJOwTLnQ/SBQCaJx176+XVEeqKe1Aaxg+5hL1pkiSz0xMHc S9Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bU2TYXI/IQ+ahWPrvtMjvIvPP35LcWsiAJFQ60YWV8o=; b=4itEQnoRkmBEJ3cZ3v7drB28+QEb+74TO/yo7i7RCPL1YQuAbgjf5slQMYCJoSZ4s4 iElCsNOgEd2UrnqAVsoneHn+GFXDgc8ZCTL0bQyn4tuBq4r3h3QF5bKkYHKCsqdV8bbR O/iwCLyp6IyD3ALJEjzj0BjXgUBMLulPQIo/Y3jp9/QnBhzp4UUAgPQCsjKGyVv5Ld6J xBPXY2HOdZEYl6N8M2ioOXJvtp5J2LmmJo7/ITdekXPSLw8OShwoi2FWTdk/ql1kgD4+ YYjCwsUG5g93pjeLrDzH2JXBcetiDTRIuK0qb9GtfhSyditfEwSCHz2Q7oUY/bs840qx V9yQ== X-Gm-Message-State: AFqh2kqak7Vg4yZ8Qi7O1/POQ+4ciIcf3FPZ/zbFP+6NmLI223HLGKDV KNQj8l1pDbMRwahjJki4TQlaWFbFASY= X-Google-Smtp-Source: AMrXdXt3ngSzeTd4BjcWtYkKZUmKlJ+coRJjsu8zn30hseAAXL6r6J79qyIJjR/uVl5OkeApgAgECG74aLI= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:9f8b:0:b0:705:cde7:2363 with SMTP id u11-20020a259f8b000000b00705cde72363mr8137319ybq.81.1673297640913; Mon, 09 Jan 2023 12:54:00 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:03 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-9-surenb@google.com> Subject: [PATCH 08/41] mm: introduce CONFIG_PER_VMA_LOCK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E13FE20015 X-Stat-Signature: mi54rua3bhdyhink1ifjrymxf1yhja3b X-HE-Tag: 1673297641-863779 X-HE-Meta: U2FsdGVkX1/CPqoyVpnzYMeEURZE0z2jingv8s+Aka2i1Zop+CTXpHADBRQV3AzDMNwFUTYO4+QcJ9ovh3crivdprnJo6CzxBIY5XLBFFwuDJUzMKbETQqn15RGH0lgiqZZwwkRXuzS4TU07sGJUlWLlkD38GYHikz+5P2pIPBpTQPQ7PZxdOwuq/x6ZlNCc84lQ1kPGOUTKtOa5zaIk7UV9U2zLY760NxPDhwkz5s8sgRq0RPyALKro5g9dzdbWqO2VeOnhAAXK4givZUIVmv38WkpkFh2f6t9iWQVrRT9fLlZggbp18T+KssFO0h+unwXyDXgRGSs16XRn5DFXH7c26J1bm9eWDbUhV6emJ9Chns3UAAyKzSvNliOAGRzXNa7Ibeufyi43N79wgOuTlnf6r6Iz1pOZ9BzqqppTSp8VsSQsineuYegksciWx/gXuiNNE+o63JHjh95CrgjazXN3XE66MJXvfX4odSpxac16fuYaN3NpsF7TZaohPlFAlc47TdAC900OJAWuOzBjSHWv/VLuO/5c2b5WIKu62xnXgJXLTLznraN/34BebvwnrjRtXrTxoNlqKeaz8tPjc0+EZq8/dJJjYdEbu4hykdiJ5Bwhj0pytiriaVlAnCyvNi2jO/yWBGAPzpwbQa5Fqbb/5n7I06G2oUIObuhmtPrAfjAJk/+mnZlNFn0TWPsEsUXLb4+bIFBlCabSaopwNcKacFA91gd4O9oixOReHUSsu1bcXS+/8MimZLU2YFyQjT+sXfU1K59xskMoaKURIVPnKWKEQFtpM74cnGiqh+vxl/jbJ2RLKX+CI1LBMMxL/PjV9EguYqWHnyhBHVDY+H6tk9pkQ5hGkCfXF70DiGaQoPY2MjEsIDgoq/vJ6NdyXHQcrXDqlJfdG/DCIyf2bcCrN/qJuOZ4TLRTKAZ7xdNyHek7+leP+k93qardLOTkphLPrtFZ0b2kDP9zh+V hVqduQoz 6obUcxXsuP0w4nfPD/KPGQreb2CVRv7v05Dc7+3IH3WFEXIXj1PXuxblK+9F8SqmdWiAocGQAjwbsOGfpxRVhnvdorXsuLL0Lc9n3uaA97hQMZBZcLRZ79kmRR4w6FXOBVQijXMBqTok32XZeIWulBhi52cFQ5BRhdp8cgs6YFdj/DSxxsLKgKRyNu6Nsjoi91AynCOnfM7bwn5A98T/2ZmK4VC/FukB8FVswMXbh5ou6fTF/sPsg47Yn95X+uhuIc1AS/Mt1uQfrSccW6JBnnOrMPNFNpCCWj8s84Xyj2ALylpmmK9gPyYjCq2jy7tQsbxhmaZhNOtm+f4T2bdkDDIQh7v2EqHpixjWc1bzCLkcqn5FWAJDxZyqaGcfAB1R4IeERNKW/zeEXRfrZ40QKlRuxOK+iWkfzOxLM X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This configuration variable will be used to build the support for VMA locking during page fault handling. This is enabled by default on supported architectures with SMP and MMU set. The architecture support is needed since the page fault handler is called from the architecture's page faulting code which needs modifications to handle faults under VMA lock. Signed-off-by: Suren Baghdasaryan --- mm/Kconfig | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ff7b209dec05..0aeca3794972 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1183,6 +1183,19 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config ARCH_SUPPORTS_PER_VMA_LOCK + def_bool n + +config PER_VMA_LOCK + bool "Per-vma locking support" + default y + depends on ARCH_SUPPORTS_PER_VMA_LOCK && MMU && SMP + help + Allow per-vma locking during page fault handling. + + This feature allows locking each virtual memory area separately when + handling page faults instead of taking mmap_lock. + source "mm/damon/Kconfig" endmenu From patchwork Mon Jan 9 20:53:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094275 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C51BC54EBD for ; Mon, 9 Jan 2023 20:54:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BEC88E000D; Mon, 9 Jan 2023 15:54:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 948688E0001; Mon, 9 Jan 2023 15:54:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E8FA8E000D; Mon, 9 Jan 2023 15:54:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6E2748E0001 for ; Mon, 9 Jan 2023 15:54:06 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2EA3D1C653A for ; Mon, 9 Jan 2023 20:54:06 +0000 (UTC) X-FDA: 80336462892.29.21E2834 Received: from mail-oa1-f74.google.com (mail-oa1-f74.google.com [209.85.160.74]) by imf19.hostedemail.com (Postfix) with ESMTP id 9BDCB1A0003 for ; Mon, 9 Jan 2023 20:54:04 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="plHU/zWr"; spf=pass (imf19.hostedemail.com: domain of 36368YwYKCP4y0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=36368YwYKCP4y0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297644; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=D1pZqRdf8jzuIQZRLrMCJmLqIEvbyFYNYfXzudRZz6Y=; b=MrKoIxhUGE5jj3UX0vqCy+H1jJ9LIJU8NJSAW3y8/KsBYOfkhUijo8JPaEmi2a6zXVAyGp DZKJcGXHEe9EW814eZfW3SVOkNRwuaKyMh8N9pSPqa3qZA+/DzoS01tVy8vYg+zqg7aBnS sTSUOSEDMjdlDRP6xx/5zCs4Mnk6U/w= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="plHU/zWr"; spf=pass (imf19.hostedemail.com: domain of 36368YwYKCP4y0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com designates 209.85.160.74 as permitted sender) smtp.mailfrom=36368YwYKCP4y0xkthmuumrk.iusrot03-ssq1giq.uxm@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297644; a=rsa-sha256; cv=none; b=5Ljgm6BOIs0bhF4eeW5bTyz6mcx45I0Wo3ORzxFQNEDQ6C+xGBo0vRoOP74UQEkWiUZeTz A5+wGZ2Vy0y1w09q2r0DaD8AxMQ36eRTWK4w8jdKbCClpoiIQBkFaF42ssw/UkeBn1uPkC SWHwTseMMVSLCt6pyBVY9zeDi8ioKIw= Received: by mail-oa1-f74.google.com with SMTP id 586e51a60fabf-15b907e71edso921527fac.17 for ; Mon, 09 Jan 2023 12:54:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=D1pZqRdf8jzuIQZRLrMCJmLqIEvbyFYNYfXzudRZz6Y=; b=plHU/zWrfFxaQ0yt/JPDRB18eEsMPdd+ZJ4E4yT8T5NRkdKBZYZu2kA11sksr3NXzt jDgkb12AloyToG1qepTp/9j3c1S/nL9/sMLfiUGj3EbCeKSxNDyG2Qbnp2R8KUfjKQNu 1hUjiCYEdhUY2xqldYVyznhCdRudSDrsopmij1IaAdQfZY5giCy+PasuJa7G7ZNci7e/ L6oXIEMKihljCOjvvq+Mko/UGqlkYlK33Q2p9mFgU8soW7h5CIdjJbM9Bn0dub4CcI+s 5hJtucAeeWPqjUkI2FPiSFxFnJ3u2uppPiS920i95ViALBCSjuucJs6UET/oQp6p8GLF Q6Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=D1pZqRdf8jzuIQZRLrMCJmLqIEvbyFYNYfXzudRZz6Y=; b=zCDhVVjT/JCCPDjL6P+XsP7FN1X2Hecti776liSHY4eYSlY0AkJKn8pNXHey7CerZo Z4zmSMiNPn7FRmp+8R1eKnNvPfRhpcJByY8y2GlL8oZv0wsnamS1q2s526EeL9cqL83W 1VEizuhYZbXQetCU597r5+qZb3UNXCvbjwEcBM/f0NW3vNbWBJxAlUupZJMSUm76JVZg f4XGnuBSTyxBESwg088pC6lVT2cSAdOi4mE5O/q4oLn1OKCgyOnEkqSGTGc8XRb84zf5 MTVgtM9qX0HchpIvDKPp4+G4o+x/xMURbNtFsPimWWzlzzBbIQHR+NnmGS035jiQ40CG 22Gw== X-Gm-Message-State: AFqh2kr1Kbg0BD7uDpXib2Rrh6aAYarJpxfH7O0rw5mEgHX8Z98NT+j5 pCZrIbicsWEMVhSGTAijEX6UK0sk8ZE= X-Google-Smtp-Source: AMrXdXv95OTi9We7BklH2FBX3pkox2UVBRMtva8Mm7ne7fTGkQBBnTiOs3huBoDZBy+tfDGkNugi4GYzEh4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a05:6870:649e:b0:143:509f:17f0 with SMTP id cz30-20020a056870649e00b00143509f17f0mr4940900oab.211.1673297643629; Mon, 09 Jan 2023 12:54:03 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:04 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-10-surenb@google.com> Subject: [PATCH 09/41] mm: rcu safe VMA freeing From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: d6oq6cnxos9ahhps4us5z3i53cn7ds5i X-Rspam-User: X-Rspamd-Queue-Id: 9BDCB1A0003 X-Rspamd-Server: rspam06 X-HE-Tag: 1673297644-380758 X-HE-Meta: U2FsdGVkX19Cl0oKHWKxQT15ggPKhO7tA2oWFWED15trX5EFMuschT/ZxZiT1kO+Eku9wr0xyDH+6egNwv55yAykKYhadKoqNIvF9PPaczzjpk7V6sgReKHqkIYSNqlkKy68nHyaECu7DrE5edxM66oIA/mj+SpWOasGia/ZXg7jA4Ha7x4TWC+0+c7smLvRMRbAIsVGKftvvKLXbeioM+ue7pNnzDMsPxTat+BsrIfQ+4W1+CBXTcA0MWD8SkmsQhI63CmwRM/fKZBIl51wn9FSY8kO/rg4RLwyMKr2kLIvNKIUTIkcncuR2S3tdf32iBcKsjLaWBqcQ+TdXwjjaYViR0Gs6owBwgjr8C+bLSZSyar5iEivMMkyiV2+P07xlg2I52ZAPgrvpJ55XGLd+C5xbvJ2Vf3aMKtna9gkUjYOYpm2i7T9v1GrEKAd9SJVEVgypHS5kHStry8Vdm+uHMzY3V+jvWYrdkBmw3tFc8VKwihA6D6ZXSsl5Jhrm57YN50SfssCEbBjMsNKELWDFT4p5hdfiF2vUDxDPUG1IoOsanzhRddW1nix0iV3iY3+8UXzfClvq7BNIXpj5Eix2PbPXRvhMeTTa1LazSdey5ic9kULOWtnzNjprMPKs1jYHd7vjuPFmGHmc4JGDSy+yuZm96pTrq4fQ59uFh/K0ja/mlNwCiHsWRU5MCy9sHNeraIBchbDzxjkYLx6L9zIzkNFZwxKJjX3q77XHfJ2MEsLlC/FMoa9GWJFdOKJcgJgoMRmp6JONks+1bIfOhVjuMn2AuVjDmmSDC/Sz6SAZGMYQzxFRhm78kyVSxRrdKr5bIhUtR7ew1viQtq1ORtq75WkbCK5CNtFKqimP7l81hs90S8unbjZh3VBJ1S5bzZKmKR/UlsDvn/4Z5IuVbRgG7SBvsZO97Tye1j2m7aLYPeoiJAqkrujEg6nTA5+7pqIsxJeggxjmy/G0yAO0WA LEZZ+pqL 8uIEy4z3VO7sW5Egdog2l8mmF711jQD2Pkya1faNtgQVeXGHUogFDhL1TM1QlTuNEQr8lRNsUaRaxTocR6LMxcfAnM4EM0uLsr8S4RsPA+rV6uEzdYsgvifHKzV9L/OCvj6TObXI6e6gu0sUV1cXNkBrtp4dXVWFhkfnYIssNSsAtBwAKSv48i3CjB1BowF1CHyMiU+W1qgN9F4RViTSysSAnig1XWtxe8uB2ndMmQuUCXFWntvkMYiG/+hCpakFLwtdUFIv5V5GVfzVcKOiN17zHiyA1w14gWB5gyeTTnyF+G50HiSSr/u/cCru4vUki4eoHG0o4dDXiZRP5h4bk1W5gemTy/nJrr9GLRIjNYl0nHoe1vpljPrGMBQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Michel Lespinasse This prepares for page faults handling under VMA lock, looking up VMAs under protection of an rcu read lock, instead of the usual mmap read lock. Signed-off-by: Michel Lespinasse Signed-off-by: Suren Baghdasaryan --- include/linux/mm_types.h | 13 ++++++++++--- kernel/fork.c | 13 +++++++++++++ 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 4b6bce73fbb4..d5cdec1314fe 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -535,9 +535,16 @@ struct anon_vma_name { struct vm_area_struct { /* The first cache line has the info for VMA tree walking. */ - unsigned long vm_start; /* Our start address within vm_mm. */ - unsigned long vm_end; /* The first byte after our end address - within vm_mm. */ + union { + struct { + /* VMA covers [vm_start; vm_end) addresses within mm */ + unsigned long vm_start; + unsigned long vm_end; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct rcu_head vm_rcu; /* Used for deferred freeing. */ +#endif + }; struct mm_struct *vm_mm; /* The address space we belong to. */ diff --git a/kernel/fork.c b/kernel/fork.c index 58aab6c889a4..5986817f393c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -479,10 +479,23 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) return new; } +#ifdef CONFIG_PER_VMA_LOCK +static void __vm_area_free(struct rcu_head *head) +{ + struct vm_area_struct *vma = container_of(head, struct vm_area_struct, + vm_rcu); + kmem_cache_free(vm_area_cachep, vma); +} +#endif + void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); +#ifdef CONFIG_PER_VMA_LOCK + call_rcu(&vma->vm_rcu, __vm_area_free); +#else kmem_cache_free(vm_area_cachep, vma); +#endif } static void account_kernel_stack(struct task_struct *tsk, int account) From patchwork Mon Jan 9 20:53:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094276 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DA55C5479D for ; Mon, 9 Jan 2023 20:54:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C0AF28E000E; Mon, 9 Jan 2023 15:54:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AF5A58E0001; Mon, 9 Jan 2023 15:54:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 996EE8E000E; Mon, 9 Jan 2023 15:54:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 89BCB8E0001 for ; Mon, 9 Jan 2023 15:54:08 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 52E1C80F30 for ; Mon, 9 Jan 2023 20:54:08 +0000 (UTC) X-FDA: 80336462976.12.124065E Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id ADCEA160011 for ; Mon, 9 Jan 2023 20:54:06 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lU92C8Jc; spf=pass (imf08.hostedemail.com: domain of 37X68YwYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=37X68YwYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297646; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S7Z9LsPdf1X6x51hAUi2Nm1VlEONIlFgbEepvCuHuj0=; b=fvDKcSCxWWN2CmVuqX1j1jzwQ/Qm+0tqGlty/CAEZDm64LItTY9yPBUKNI9w/UqTwryCJY IjUilRbFayb9Kkcz7V7PCLqwuwUww3uHrEJpkL2VVhvzzkSQfuSAbeCBULuxcrzZSXaXtj JXgEy7wR/l6hxn+AdB5GR3+fayYW+jo= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=lU92C8Jc; spf=pass (imf08.hostedemail.com: domain of 37X68YwYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=37X68YwYKCAIuwtgpdiqqing.eqonkpwz-oomxcem.qti@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297646; a=rsa-sha256; cv=none; b=j9ZwJNCXqH+CtQPf2Dd7O19Ke9wBLPVadoOZQnbtVceowTSl9dVMbbt4nOPvDUWckwyXft 4PbqQVEP1lYl/ofBk2Uwlm3jr8hmR88HB6ifXUnq2YaNg3wSCmGBFpqYm/ttNyALflhG0B 8S/vz+3iCnpQeFNp8udVzYrpys73M1Q= Received: by mail-yb1-f201.google.com with SMTP id e12-20020a25500c000000b007b48c520262so10313453ybb.14 for ; Mon, 09 Jan 2023 12:54:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=S7Z9LsPdf1X6x51hAUi2Nm1VlEONIlFgbEepvCuHuj0=; b=lU92C8Jcfmul8qFoqZuzCOb83nRDo2DOHABHtbUeff7KpjabsQdodVMIjgjNegm4Ui gxUyuAPRcETsiRyXJ5mliOj6g1oYM6B1qnPvTAT1LA3dK82Ds4hQfkrM9RtYlbde0cbx i9uLWTSQCuYKj1fxgTqem2OIhsK37AMmmELynCklGF8pD//eviNoFDjhffy4KqRRdQJC MSzyVPeVUGI3nBlo70SJ15d8SfW3UoSbvzQXr0uydmfUdXg4Fl9jpjR3RSUiLlvME5Kv Z62tI5fReQIg+DMNczYR66QAZbTIG8pGdwhVnCRBmPZaWsdfRkDIb3vX4UMeFSba2+4A TcUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=S7Z9LsPdf1X6x51hAUi2Nm1VlEONIlFgbEepvCuHuj0=; b=phhJFKXmO7qMJ5/P+TEpiZGW/BKgRn1j6IzUwZqJUpIkPiQf0puYDgAma35HpEWEl/ btgsrmtNZZdu5IPako2pf43ZzFStH5SvcI1c9Srripbi7ncTwGKgwRZ7m2/HhSNKaZSC h2kEa1rQBHZnN+VYdjUDhKZqq4eLhO1dhjKQo1HMIwka2fb9I8qHKjiDYPjNsZPu8gTN 7c0wCqVp4Lsrb7XWSF92VIH2ot1cU9A0SLNl7dZ6LUauViEQXTVVh8u8EU3ozugQQe4+ UNwsXk5cfko2l34Ab4VlJyobWq3EeODfx+YAN7/hJg93iQR8C9Rbg4VwQhqSOAsBikcI +dgg== X-Gm-Message-State: AFqh2kr791bo6/UxgduWSmNFcYpOlo2Ym7Hkf0fZDOVsPh3hdE0ZvWFr naQnPP1OEgpd6L6eJIqwB+IA/+byTC0= X-Google-Smtp-Source: AMrXdXt5VtatAvHvgu6DDkpLTQLv6tox4QHYgsFVwcGGt6mUkf2zF94ad2ULjvwhvY9gJJrtYWAmbQIwTHs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:4fd0:0:b0:468:5f78:265a with SMTP id d199-20020a814fd0000000b004685f78265amr1632391ywb.87.1673297645909; Mon, 09 Jan 2023 12:54:05 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:05 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-11-surenb@google.com> Subject: [PATCH 10/41] mm: move mmap_lock assert function definitions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: jm5hkip9atew5zcfamfnx3ezxne7awa9 X-Rspam-User: X-Rspamd-Queue-Id: ADCEA160011 X-Rspamd-Server: rspam06 X-HE-Tag: 1673297646-576051 X-HE-Meta: U2FsdGVkX183kfp6gszFr7rgKRxS88OMAbNi9gVedJxjvm53djdmVeKgNv3eiozLWWL7LgOOu5ySNJtxNw8bIUNBdFx2dtF26P4EKUCSrkgf4U5AUqUhxYdlaU6jbzx4fa10s/vlt/Bt0DN9Ms7UK1d9xWS/VUE/f0qMXy0EdYCwOfEj92yEYfjyMyawp+Q7RwumE1Y9P7FfhcZMEwRfvhnDcmf7d16tXLDrZONl83exp46q7as3gSYLfM++X1AhKI4po66eVbLNyO0zbt8yyVHxwLNr+9Mq0NzBkcP5EmNq0qSl43UenwV7Hz5ReIFAhYXZ32ig8BJjvGEbqa/KdafhXp3S9u4j4yJaguELXx+1daNt+TnNbbnXSBFRwrV9J36wA9uHgUZTZcoQYSxLkTi04ckcA4vKEW+Gh/tPG0BKjDX94d1lG6QWu5PUvMFeAlnMJkifghRQl1L7nGkkSUtDJbxiwbTPHRWBbKOIEV85ui0O4ymYMmjcBVNXz56n6y8R+GnxjMmfuPosB2NGB9ZfHQ6mQncQyASQj9JmEsgKF6+Mt0/lHfrFK8W2Utqw36uv1d91zZp9XztX4jFaDEsLpIKC/9cMRyEugMi4+0P3zaF1nAaRJqU6/wArtc1TXY2DwxtbZ0CyVk4xyd3WKP1Q+NFtVT7hCwTk/K6W1l7m/8f7mQd6yaY3+2Wz9Z20OFPJA9M4PaI4VtHdrniHF05vNjyLhXiP/QJM5AsDizLg14ZKF7TlEO7vKRK1QJldgNXQ9L1NHby5T50LXT83Ob2VHMeXoyiaTkL+X+NP96HmMEKoxOJ5ukUMrohG8kkVpsRcp5XhQAwyCIe1+RzuTgoLK3yCyoL6qrkrIng5rXG5pLztDiNTLscMMyuI2EBBqD9od4Y2bh9kvN9IUNcWdkdlf9bIuVuLWoGJo8bawqLjXtG5ibQMU1FTDYAx0pvv47q4fXCaszkmmIHmMaO Cid3nbuZ cC03RvD9Tlh87CioLkCb6n0ceKfn9JpJPMyOpf4J6Rjr/wsXY12arxgY9WYBvrkFgTyRn17w07e63v9nyfLtT+mIooM9cM53ttC1fyqjteW/Io5yh6fh5y07rEIH+NT+oa834JL27SmZsaJeVcAjsSMxK2MVD9Fw/L1jCOBI8dEOOV8lR2V+8QM8ndWlLnsgYRv/gTEuKE7mRO/3REtlqP960qBngeN24k6IgVa/uiI86FCUFdlTnRSm6XStb6QjcWhOmrNvX1JOVGmo0VwYbgfpis8DhbLLqWWG95OyBrfGzg/ApMksmbTlrlEpxdGD+aAIZognXUawOAgDyYfNN7D/eRD5U5lhh1EorezhbNaNdSMf5ZQ2JtBGUsaduUWwKMRu7XaJwdTruYtEFv60xEU5T+0MsVqBNuh3o X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move mmap_lock assert function definitions up so that they can be used by other mmap_lock routines. Signed-off-by: Suren Baghdasaryan --- include/linux/mmap_lock.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index 96e113e23d04..e49ba91bb1f0 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -60,6 +60,18 @@ static inline void __mmap_lock_trace_released(struct mm_struct *mm, bool write) #endif /* CONFIG_TRACING */ +static inline void mmap_assert_locked(struct mm_struct *mm) +{ + lockdep_assert_held(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + +static inline void mmap_assert_write_locked(struct mm_struct *mm) +{ + lockdep_assert_held_write(&mm->mmap_lock); + VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); +} + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -150,18 +162,6 @@ static inline void mmap_read_unlock_non_owner(struct mm_struct *mm) up_read_non_owner(&mm->mmap_lock); } -static inline void mmap_assert_locked(struct mm_struct *mm) -{ - lockdep_assert_held(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - -static inline void mmap_assert_write_locked(struct mm_struct *mm) -{ - lockdep_assert_held_write(&mm->mmap_lock); - VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); -} - static inline int mmap_lock_is_contended(struct mm_struct *mm) { return rwsem_is_contended(&mm->mmap_lock); From patchwork Mon Jan 9 20:53:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094277 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52920C54EBD for ; Mon, 9 Jan 2023 20:54:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2CF88E000F; Mon, 9 Jan 2023 15:54:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E032D8E0001; Mon, 9 Jan 2023 15:54:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA4858E000F; Mon, 9 Jan 2023 15:54:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id BC0CA8E0001 for ; Mon, 9 Jan 2023 15:54:10 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 90461120C11 for ; Mon, 9 Jan 2023 20:54:10 +0000 (UTC) X-FDA: 80336463060.01.3827950 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf08.hostedemail.com (Postfix) with ESMTP id DEB4D160006 for ; Mon, 9 Jan 2023 20:54:08 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GGWOlVFN; spf=pass (imf08.hostedemail.com: domain of 37368YwYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=37368YwYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297648; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VKcEnbcUm4yNPmJ0/UBWAk4YOoOktsWvie1GPNgM1C8=; b=BXdX0xKgucf2+3V/4ptuInSVIZ4YT6nDupN4R1YekB5mPdF0F/ugRioz2Owwnzh57lYjiO X/9cYY1LN/cRA9qZIi5gZ+bkzgJwPlZRjudbAYqZFBF1ao1QKotmdp0sw+fAgSUVFPYva3 Y8Nw/xioq2O/4F5fbrjKiAG5MhcT45E= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GGWOlVFN; spf=pass (imf08.hostedemail.com: domain of 37368YwYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=37368YwYKCAQwyvirfksskpi.gsqpmry1-qqozego.svk@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297648; a=rsa-sha256; cv=none; b=Kj0cN0221x7zys0V6QmRO53mAKS/CGRDFzlxH+2DsYWxbM8CUSlWoIH+D+fm3weeLgMImT 7fqT+MYfxqt/B1drdQLTlPilvjtoew/7a9+UNLzabLTWcHKKDXunsn+nVPsXcioDGGi+qW xSVqenxT81PfFmysE9dKwGxKaec5764= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-46839d9ca5dso104888827b3.16 for ; Mon, 09 Jan 2023 12:54:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VKcEnbcUm4yNPmJ0/UBWAk4YOoOktsWvie1GPNgM1C8=; b=GGWOlVFNZmpsv4EqX6fJ0IzU46mRQdwbpB1woCBvprw4wrB1Px0W3aw0R3eA4F/+xV kXuBRDFymMvypEzFr/D4iD7og2WAH2DpqMoKxtl2yVGo8HrvBlCIzI0TYQCav4I1owI5 QhJAem5dniKSevuuTKEF0EwC81wZP9AQ1hwH/aOaMvCSNfv3FcYZaW7nXgrr4EsyFNJE AJznojM8o2oQLC4YAgQdb7K64EDnrtRV2QjiyePckmOOYm4vpeviEyBwwk5GM5l0mBi5 NKaa6vy3INpTQuuxZ0BRwKOg99o3TKTitzAEr6jS2DiN0O+IImG4Tiab8jZ8nkO0QP6Q MsPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VKcEnbcUm4yNPmJ0/UBWAk4YOoOktsWvie1GPNgM1C8=; b=pKrxMc/B3Wipr9uVvv1DrzkQlX4a0WOkmZI44Y1kJalOUHbO9ZzA2boYbFZUjzTBsY nT5xQo2vvMO5FVv9uv4jAUkTSPZN4dVa5OlE2TYKVQ9V0mPQKVblPA3vHmThlLlUwhLG 1wrRdM8+PaMw7xUKh9/ssqURWxst2HV1nzgcHW5XFf0lH9PLvhCaNCWb/X5eilgf0W0M G3Puwr97fZ09ogYYFS1jLc8kGR9E+gPMJPJVRMMjXAUR8Lx6YQnNXMPwIs2xCrznzWRS bYsYpGvkQDK++vOTtviJWepRKXCe0v+apB+pRMNIXALics27uZdhK9+0HYMKKD86ZZ0h Cipg== X-Gm-Message-State: AFqh2krMMYiezS9lQeHA+dUV9EMuwGketWBgO2cNP6BneqoJk8gRGMrf TPqzPGK4C1qCyaBayMF5rhorj1eCoYo= X-Google-Smtp-Source: AMrXdXuuklmw1U5OSS04yToFIKYXQnoMsaTWu76HfgvyMhLNzfJxTKm5wgWz/6XkfwWL03n+CBAO8y2WRQ4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:d48a:0:b0:46a:f5ee:a2fe with SMTP id w132-20020a0dd48a000000b0046af5eea2femr1229443ywd.207.1673297647974; Mon, 09 Jan 2023 12:54:07 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:06 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-12-surenb@google.com> Subject: [PATCH 11/41] mm: export dump_mm() From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: 4ywmcdahkgck1uk87rke4athsd3ow7rb X-Rspam-User: X-Rspamd-Queue-Id: DEB4D160006 X-Rspamd-Server: rspam06 X-HE-Tag: 1673297648-439662 X-HE-Meta: U2FsdGVkX19HmiHH01uVpwmXdax5MEM4Dw0ECVGke4OdBjiRZtcORhbnu42qSuxPBC3ZqLCBjtACV0Nrg+2k6SUleSG9VC8nmGYX+VDf76GVBrTaP2voIYOE2AoACS7Wwexnx+ThJ7JW1MD7FKJiaYQKGKQb2JGQMOj5nl/XWdMNSey0E8Faldcem2Qj81JsDlezHOa7z9V9A60YMVjnQQyTAQ0WCWuJ6xXB6uUHwI40J1NUmxgH9cm1jaSMSIqOrWVf8Rj7hqpP/h2JKjVgKfxJlo4Zg0Alu3IZ9p/z/rc+UjytXNefvvKI5OdHCYrLy5Wex3UT1WqgDPbCQwGbU2uBT2awCSs+rBMhAmvnOOo7uawo3yWOY1MjfeeEWXc8XDcZBXxZsqlw6GYWD7JO5dnqo7adeHo9CcUjTcvJYc342U2p3LlP5cGqJH42MBaEfV5yK3foyI38xmxMt3w5EG6+I6F4q9hJEI/xq0W16Is1nURDNo0X+xyc3T0vhDDCJBLvQPTrHGEjwMNOmWzRMPp4LTYKJghjmxr92vjOfBMmKzU6lQwclcFiv7zDkmdfFiCtj8+HD4DFwKkvjomX/Hk+VfwmWvjqhxZYpIgSsr8rb0Ee+PrcLo4V3bqrVnW0bJzpQjgOZLHtqJVYfZweutRalDHkZB2dRCP0WcVNRYNUO5KPrYS+iQJFyYeuW0d5iw8d/CQw0dDuCqntBdDfmdB2VWYkIIM9jwbctTlcBP194EiUEDlKUt4RiSnmJr5ctxSBvg39uhGC3b7qspv8egvKrhU3ZcBNjQpcJIH/q7dIat9ee/qDjuya5JlDM8fre4H8P6OjgrhSRgDZwq8yJjjFxMu3rg4GsXVOa/zOO1zMC3hy8U58Xy676EVezIKlyci7i+ZIWgh2rP/C8AH2fQ1xh/P9HUg1F/0Q9smYvNLcnHE06X5ydZhWgqfI3n08Nz5UsZrOPRI441pxcUj taJC5E4V UjCOi0uqw+eLwGKFOjXDp6QXATmF2sGoYKaVGOh2OzuCIj2YMADuRzvF9JwYXb/YUny2fJFJHbjX7Q/u7m4PdxfwtKS2MtHjM092CeD9JRSyFJBowgH/Mrp80LU7Uv9ZJC3EXkQr3/yhgkjbuCdEa9KN+4cHWvVeL0pxKkcb4hJduJ+sQ9LDeKxcC1/jp0fL/ahTo07QFPLb1LWLiP8j2oyFqQiF3l5YpBllyXKxUm7/crkTIbgXteGT0H2SinNjEPSl3J7/fMYr/eFT8kN003simBdan6km0/r2iy91lmVrkHOrt6phNEo0FiGpjd+xWuJ5lfXDLp2UPRYxW/4UfBlflnVw85nWReC7Kzv7TPQkE9n7YbxMTqs76+Xf569NxefMQV9q0kscpYVZBzRshLGbmgdz2Eo9hv5q0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: mmap_assert_write_locked() will be used in the next patch to ensure vma write lock is taken only under mmap_lock exclusive lock. Because mmap_assert_write_locked() uses dump_mm() and there are cases when vma write lock is taken from inside a module, it's necessary to export dump_mm() function. Signed-off-by: Suren Baghdasaryan --- mm/debug.c | 1 + 1 file changed, 1 insertion(+) diff --git a/mm/debug.c b/mm/debug.c index 7f8e5f744e42..b6e9e53469d1 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -215,6 +215,7 @@ void dump_mm(const struct mm_struct *mm) mm->def_flags, &mm->def_flags ); } +EXPORT_SYMBOL(dump_mm); static bool page_init_poisoning __read_mostly = true; From patchwork Mon Jan 9 20:53:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 151DAC54EBD for ; Mon, 9 Jan 2023 20:54:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA9108E0010; Mon, 9 Jan 2023 15:54:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A59C28E0001; Mon, 9 Jan 2023 15:54:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FAA78E0010; Mon, 9 Jan 2023 15:54:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 80A1B8E0001 for ; Mon, 9 Jan 2023 15:54:13 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6407840F44 for ; Mon, 9 Jan 2023 20:54:13 +0000 (UTC) X-FDA: 80336463186.10.4DF2CCD Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf19.hostedemail.com (Postfix) with ESMTP id D270C1A000E for ; Mon, 9 Jan 2023 20:54:11 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TUshLIkn; spf=pass (imf19.hostedemail.com: domain of 38368YwYKCAg02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=38368YwYKCAg02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297651; a=rsa-sha256; cv=none; b=d0iv2wGL3Liu1sdhAw5qa2wfjfW/2cyft5HMaA18TBho+VlOFuz+bB474mHdPt0r41CiHj q0aLqS7HTG7iJ8wuac5Gtp2Sytdd6MaOJbxJlcx+vW21hpZTjEeY4brIIez16xVlZxjqwU GNSQrzBBplQf3SirkIGt3U4qNVw++6w= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=TUshLIkn; spf=pass (imf19.hostedemail.com: domain of 38368YwYKCAg02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=38368YwYKCAg02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297651; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=79bvrukKvLAS13wwuEGJsW+4aNck296i4OOdN8AZrSw=; b=VeMdRzZbpWfHcTK6cJXIGDT0sl4i+QP1yrJ5VSZJZbWM9zDgrMfO6b7+FVw99mBfSlmG+5 8eYt0XOSPXAdTPR5/QdItTPW2J4MyI/EMt7R2ldf1Pn6W882bmr994VFZ50K1HqR6fhFyJ MGLsQ7Gy8fs8kDI2FaY9/KyA/pT51MA= Received: by mail-yb1-f202.google.com with SMTP id l194-20020a2525cb000000b007b411fbdc13so10389429ybl.23 for ; Mon, 09 Jan 2023 12:54:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=79bvrukKvLAS13wwuEGJsW+4aNck296i4OOdN8AZrSw=; b=TUshLIkna8lEowgf738l64K6OhURrpjePFxYxdKUuWhyUGH2D62dGB+OGRki5oXp4C xTYuRmR7XzQ7I/28ieatH6mJigev2F3K0HIzC6hflnBGI34WrRdJkJZ4AZfQFOQ8gGav 9UNpZpO2F0owMoX4KYmO5GlLvBicz0eIhvCyoFVAlBVkJOLooeK67cBYtsiWvskUp6uE tE8t31WtIP9ueuHL/u/lXs1s/RC2HGn9FvMQfHmx5Itoy4F3Nj/5Ls77RI1e7g50JCXC 9+/P6cV1602qOnEnSC6h6SSpk6hPN1ukv5Ai24vZZWfGTttvD7F17H6z929cULqI8Sm7 oWIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=79bvrukKvLAS13wwuEGJsW+4aNck296i4OOdN8AZrSw=; b=tmzsCP5ujuODcCJrqKhZhZzt5RlItmyXKxGDVXHbYREHQaL8HnWFrQTKF+1/ywksKe O6VEUaLzjRvQVMEHh0VAJ0K9uArHmMT8cCEikObV5TohYxAuq6Fru6PSetSmqZYAfwwn dyvRyTDsGhjHs2KtlzrUIo/kwN/nV+z1d/On7ybRr6Knyc7RAK3AjbkH3buEZRYfpbtP 1neGI0UsmLCoDQLC4gzU/AicF1PHLR7QK0VYqn+fk0i/WCN1A95CtCoNiQhrm3PK1goi UXDQ6brycIHeRPk1RJEC12cx6qhbJJy+b1yxZLZQsvgeEkMI/K4pMRvrA7oVtrR6GQWS u1SA== X-Gm-Message-State: AFqh2koCV9aJcXINjCj8oAAqoBJhhg+Mz9l7DwAb2RBFTDHs2k7ZO+Ff zzFNPpEbAsFeZaLg9ZwO005s5ZIpYeQ= X-Google-Smtp-Source: AMrXdXukETqaSxCpG7iZuCRz7oEpYHbdWSzbf1+vDAfX24FIika8R4beORIYrEIIAWwV84R+bp3+gXy0ol4= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:c481:0:b0:4ad:7104:1f66 with SMTP id g123-20020a0dc481000000b004ad71041f66mr3418570ywd.53.1673297651084; Mon, 09 Jan 2023 12:54:11 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:07 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-13-surenb@google.com> Subject: [PATCH 12/41] mm: add per-VMA lock and helper functions to control it From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D270C1A000E X-Stat-Signature: 1gdtqbxaa4ofcpdkwm877rg3kkawe74w X-HE-Tag: 1673297651-451644 X-HE-Meta: U2FsdGVkX19FAeAwNlsbbGZHwM3MvxNVbvd7QwVLgRO5M+qBSWaLGloUGi234e1vzwIT7kOUzKLPkXUeyax2WR4gS4X0S/m7g6rf44ou3uqwAiXtsHi9W0yAacCRZSvMwrxMj/b9vxzm3OSXnoaTP+MQjYi2uv2DQpSTyWErIwflXw5zYQcXzA+lQ1wdX43L5ImfBZSf9vyLgn+0diquFnonfr7qFBXRxW6k89BKHGnh5p8hUzvfjaHziNVuEfNdRyhOiVjmU1dRNxCBz0RsqNgBPQoZP3olOVn0gdeg64+54Iz/OTjodpGFPs0SorkLUvFig6lLo88/c+Q1o/EaS+H29Yn3O/A2YN86d6U8EcOy1RM8QW1dFalTksFBY19AumvvxIjb9VpdSwlERHVHrRMrZfwuiianwb/+sU5+6CPK014NVhLueVynN1PdKaoEeFoKIfm2DzlnkbQEVkb3UgYiQEyCWYkltgnEtvYju00Yd3jvvMBkh0GXBBdl/ds7HuZkcnfOrk2HvRilL/6735Qp70k03muHLYsgyHEvYl8BxbAl/AKoXrV/OPF0ffd0nTHM0/COHzTsXWiT1uRwUey6F73Ky17emdk5IWaWS8Hop60UtrFHZCXLB1gTLJbXPYs/RJiH8wGjaKDRiQW0lOky8/13c2DpBaU7Xrgrm3o9nGeIpk8PbjvdTrBpcDFLWW9hJTsd4zv05K4xrxgUVjyfCc9vckUKD11QA+FcnaAffQRl36t/B565/N7c05TDyng9qI7zEnCnaptBl+XMl4qAlnyP0h0wkdE8Pr/KmdNawExEGNY+32Dup6S9fAYtER8WcenNvoDsMTDRH11VwpjoWUI+22CLOB0ucS9dZshI00oORD/VhT19wB+PshkApaUipOr1MBCBgjgLcJHImBvNXOP8MMRkhCGbhGlTGaY3QHrl4O4jxVRb5tKorLgaV4B/TqejpNnDfPFFHfI JdgIwWgV Kv/DUA/17fVrFpqAg230zHh7YvfdXe5sTzxdKNPwU6eqiHh5bV2h6D/BqRuPRQccZ8dPmpRYnJM5rjE/7PYnNoqfMmlF1qNx++9FCxzMcVgKjfR1DyQJjdG1shgT/gKoyRfDk/r3AOELMl+YEbaHh1uFAg16DlZkNzkMlWL8pSnvckKN7wnkMwGTXbPKfvJws/00kbQLB9u4sz4lq75mcOtU+BVRrBBmIUhgjY0o3xhBm/az03m7sZR2kZIpfIOx3UF2apP8dCZf8mFyC6x9xY+Pn6BjqInmeXhYkpZGBDHOEFyqPI3wq4QZGZymem6/sE16e7o48fdCCvnt2WtvbPoMugByBylSgBnJOX775U/CUsk1SmVK9g/DZGg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce a per-VMA rw_semaphore to be used during page fault handling instead of mmap_lock. Because there are cases when multiple VMAs need to be exclusively locked during VMA tree modifications, instead of the usual lock/unlock patter we mark a VMA as locked by taking per-VMA lock exclusively and setting vma->lock_seq to the current mm->lock_seq. When mmap_write_lock holder is done with all modifications and drops mmap_lock, it will increment mm->lock_seq, effectively unlocking all VMAs marked as locked. VMA lock is placed on the cache line boundary so that its 'count' field falls into the first cache line while the rest of the fields fall into the second cache line. This lets the 'count' field to be cached with other frequently accessed fields and used quickly in uncontended case while 'owner' and other fields used in the contended case will not invalidate the first cache line while waiting on the lock. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 80 +++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 ++++ include/linux/mmap_lock.h | 13 +++++++ kernel/fork.c | 4 ++ mm/init-mm.c | 3 ++ 5 files changed, 108 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f196e4d66d..ec2c4c227d51 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -612,6 +612,85 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_init_lock(struct vm_area_struct *vma) +{ + init_rwsem(&vma->lock); + vma->vm_lock_seq = -1; +} + +static inline void vma_write_lock(struct vm_area_struct *vma) +{ + int mm_lock_seq; + + mmap_assert_write_locked(vma->vm_mm); + + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); + if (vma->vm_lock_seq == mm_lock_seq) + return; + + down_write(&vma->lock); + vma->vm_lock_seq = mm_lock_seq; + up_write(&vma->lock); +} + +/* + * Try to read-lock a vma. The function is allowed to occasionally yield false + * locked result to avoid performance overhead, in which case we fall back to + * using mmap_lock. The function should never yield false unlocked result. + */ +static inline bool vma_read_trylock(struct vm_area_struct *vma) +{ + /* Check before locking. A race might cause false locked result. */ + if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + return false; + + if (unlikely(down_read_trylock(&vma->lock) == 0)) + return false; + + /* + * Overflow might produce false locked result. + * False unlocked result is impossible because we modify and check + * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * modification invalidates all existing locks. + */ + if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + up_read(&vma->lock); + return false; + } + return true; +} + +static inline void vma_read_unlock(struct vm_area_struct *vma) +{ + up_read(&vma->lock); +} + +static inline void vma_assert_write_locked(struct vm_area_struct *vma) +{ + mmap_assert_write_locked(vma->vm_mm); + /* + * current task is holding mmap_write_lock, both vma->vm_lock_seq and + * mm->mm_lock_seq can't be concurrently modified. + */ + VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +static inline void vma_init_lock(struct vm_area_struct *vma) {} +static inline void vma_write_lock(struct vm_area_struct *vma) {} +static inline bool vma_read_trylock(struct vm_area_struct *vma) + { return false; } +static inline void vma_read_unlock(struct vm_area_struct *vma) {} +static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} + +#endif /* CONFIG_PER_VMA_LOCK */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -620,6 +699,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_init_lock(vma); } static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index d5cdec1314fe..5f7c5ca89931 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -555,6 +555,11 @@ struct vm_area_struct { pgprot_t vm_page_prot; unsigned long vm_flags; /* Flags, see mm.h. */ +#ifdef CONFIG_PER_VMA_LOCK + int vm_lock_seq; + struct rw_semaphore lock; +#endif + /* * For areas with an address space and backing store, * linkage into the address_space->i_mmap interval tree. @@ -680,6 +685,9 @@ struct mm_struct { * init_mm.mmlist, and are protected * by mmlist_lock */ +#ifdef CONFIG_PER_VMA_LOCK + int mm_lock_seq; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h index e49ba91bb1f0..40facd4c398b 100644 --- a/include/linux/mmap_lock.h +++ b/include/linux/mmap_lock.h @@ -72,6 +72,17 @@ static inline void mmap_assert_write_locked(struct mm_struct *mm) VM_BUG_ON_MM(!rwsem_is_locked(&mm->mmap_lock), mm); } +#ifdef CONFIG_PER_VMA_LOCK +static inline void vma_write_unlock_mm(struct mm_struct *mm) +{ + mmap_assert_write_locked(mm); + /* No races during update due to exclusive mmap_lock being held */ + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1); +} +#else +static inline void vma_write_unlock_mm(struct mm_struct *mm) {} +#endif + static inline void mmap_init_lock(struct mm_struct *mm) { init_rwsem(&mm->mmap_lock); @@ -114,12 +125,14 @@ static inline bool mmap_write_trylock(struct mm_struct *mm) static inline void mmap_write_unlock(struct mm_struct *mm) { __mmap_lock_trace_released(mm, true); + vma_write_unlock_mm(mm); up_write(&mm->mmap_lock); } static inline void mmap_write_downgrade(struct mm_struct *mm) { __mmap_lock_trace_acquire_returned(mm, false, true); + vma_write_unlock_mm(mm); downgrade_write(&mm->mmap_lock); } diff --git a/kernel/fork.c b/kernel/fork.c index 5986817f393c..c026d75108b3 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ *new = data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_init_lock(new); dup_anon_vma_name(orig, new); } return new; @@ -1145,6 +1146,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, seqcount_init(&mm->write_protect_seq); mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); +#ifdef CONFIG_PER_VMA_LOCK + WRITE_ONCE(mm->mm_lock_seq, 0); +#endif mm_pgtables_bytes_init(mm); mm->map_count = 0; mm->locked_vm = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index c9327abb771c..33269314e060 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -37,6 +37,9 @@ struct mm_struct init_mm = { .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock), .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), +#ifdef CONFIG_PER_VMA_LOCK + .mm_lock_seq = 0, +#endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, #ifdef CONFIG_IOMMU_SVA From patchwork Mon Jan 9 20:53:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63BAAC54EBD for ; Mon, 9 Jan 2023 20:54:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 021F48E0011; Mon, 9 Jan 2023 15:54:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F146A8E0001; Mon, 9 Jan 2023 15:54:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8E348E0011; Mon, 9 Jan 2023 15:54:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C96D08E0001 for ; Mon, 9 Jan 2023 15:54:16 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id AB8C4A07C4 for ; Mon, 9 Jan 2023 20:54:16 +0000 (UTC) X-FDA: 80336463312.23.B137443 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf01.hostedemail.com (Postfix) with ESMTP id BF0F340018 for ; Mon, 9 Jan 2023 20:54:14 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="RKn/zVLZ"; spf=pass (imf01.hostedemail.com: domain of 39X68YwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=39X68YwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297654; a=rsa-sha256; cv=none; b=W5PK8uK3UxFqqfCtjs6qZ/YpUSTk+2SMsOig4bTYCi6DcldZZ/wjiNFy2MCBQojvQj7SA9 bWyPIM8jA/TtfUZyQAXBokxHuGzknwsS9uY97ajpvHFPOwRcNSGazCKBdkF4VvISF3SRcp xBwjU2lPp5b6zcv2PMVdlp1F9r8g10M= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="RKn/zVLZ"; spf=pass (imf01.hostedemail.com: domain of 39X68YwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=39X68YwYKCAo241oxlqyyqvo.mywvsx47-wwu5kmu.y1q@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297654; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7ThjMRKbwnKxVGB3eylsidymuk6Qi/iLA6nYtJEm4MU=; b=PcDpaHXqBDcPKUtB2+YwE+plrd5lkPIx7nH3CLwwHOh1b/kH5bKUgxIxUHAR084h8RZVs3 lm3OZYz/DTqAnWoBe5k6UPpX/P0pwVk9lTPNwLE/72UwpwNcHFoDo+DvjEKJ/Xww/wCX7B +YzDxHOrXo+6la5K9+AnKYt8qAzAoD0= Received: by mail-pj1-f73.google.com with SMTP id t12-20020a17090a3e4c00b00225cb4e761cso3632327pjm.3 for ; Mon, 09 Jan 2023 12:54:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7ThjMRKbwnKxVGB3eylsidymuk6Qi/iLA6nYtJEm4MU=; b=RKn/zVLZEJYWNukTXjRCu4nz81zaytlRkR7rakIR2uB/aIfeElp0yCIRzuM8+pvZ3x BQOElK8tbS+rhMmhhEhBtfHOWGZOyvH/HCUdLENpEngLYbZsqjJ9OUXahFZPUHh+XaxU YMwkVhfHQ73TasfAveK5V4w8pX7O6IKpRzfg+l1/+SN9l2l60gXMRIOKbNPAPi3/90FE v/HorT/klJnq8Lcd3XX+VkW89j017v/vCfpCQTBG4XFodaDoUu6obKl8t1y+jpshNDo0 91Vfmkl3EHo0Elu4jQ9efLprq8Gy+cE160O08y2PdqRoSx1lsEPlDSvoNxeAyXeTk2x0 qpkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7ThjMRKbwnKxVGB3eylsidymuk6Qi/iLA6nYtJEm4MU=; b=OOBdR15WHsIwGvBIwPOO+uhJZRM1V1ACOAqyeAudP5jLAZjd2fu8El7LqCsZt/BiJG cAQcSqjfP1ABG04SvLWeJFgbJw1SG/p10ucpMNI/qn4Zp1Qgft4KkqOa2ld0cU+DZCw/ fVRagdwMH3Z2LQHYoZRv7ZCmmw+0XvU80tXzW1JLi/uMeeTYUNZ7DYLQGmeyUTdaT29X Lq7HaphQd3FKI2DrrmE7f1vZ301aZnsWetuiiQSnQQgW3H17XjBAaZHm8VTOjF4krASw PvJRHmY24nMflWkXa7MtWi5sgjguseIK/aXUo4ivI/uBZnVQYcoFgBYLLrZHUU9Eny3r N3Yw== X-Gm-Message-State: AFqh2krOhgphpp/DFbtdq7cPeXHzLdDZV6On2SouiWuUxu0c/a5sMDUJ aXqVYLHPKAy7tjE7wWMyxnap9/yoDuI= X-Google-Smtp-Source: AMrXdXvxnut8BEB8WYWMFHzY9PnfOwu60Xd9fTsfu57bMChvZCCQkV4pslftmJOaEGGsy1IoVxboiXkLZWU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:90a:5c86:b0:219:c1fb:5da8 with SMTP id r6-20020a17090a5c8600b00219c1fb5da8mr5399379pji.221.1673297653600; Mon, 09 Jan 2023 12:54:13 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:08 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-14-surenb@google.com> Subject: [PATCH 13/41] mm: introduce vma->vm_flags modifier functions From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: BF0F340018 X-Stat-Signature: t5e53qpo5su6rd41d9hpos1e8mhjyrus X-HE-Tag: 1673297654-194642 X-HE-Meta: U2FsdGVkX1+BGDaJMqOYj5Wq5k29hrgRPqUAAH/XtdVF5njfIm8M/UW3g+LdwscCIayUskDvKm/apkq/U0dM9jzkQhVbd0jf98EC3VEFtiv2gRfl8ISlSI8oQRCTZKf7smSofgI0JUV6l47ByRS6ZcHnF2G/oT2j/K7ZVZqho2MTb51vxYeew8PsSkSm8GmuSbsNim/34aorIUFx/NDhXy0PSMwu0vs8Kpa3aIAPxbWPJlvfBpHO9GVVzv7Nps46bgN02yEtcrahfiFZ4WLFdpx7CVZx4Cy4sndHAKoqtOGObIHVCDUUXGzUceeBC/pl3UFjMTFGxixICDAbYZLg/R8KZTJAW+hk6VP3H+j73mT7Gs9C+vuoRkZwTe1aq1Co4HM8RVUwsD6H+3mdG/cWgOS9nP+xA+nSsvTMTidLUawJpXcBYbHB3I7qaZK6YIcmh+ZZn5QgDUbAMZzJ2IDOx7Grg7fsJ0hq4R5hTZEWplGaGiNj/+vmo0u8+l4cOLD9IiU5BDti9yLP9FdZ6ifgu6AuzkaZGAxidu3wPkbR1KJA2MKRzUZ6NVhD0l+3PvAwmSd8VtfrFDUuSlmPRKoaN10rM39krvidvymKAsEuDhEmbio3LY7HdlNOmCGmdiI0/g9SFDEljY9JsZ1xPwOItm9tsI8usb83L2iqc7zNj9ySh2R3MgrlPEmzvjLeE1Te6VuOEbOcHat8htGplS/84kuxKxQhaF1/+zJmWD/BDTvfzl2ClWduGhJiIDhaje1kuxKla6Xsp3cNpt5vf1euz6iz06/1E2v47C6LAzrPS9w3Cz+a+XvsrX/+ZvyxMfibJ1pNtFnxyjFjO4yIfsLFn+q/B53YUql2gOgIbzJZKsC0J1kbo4bpFAzhRP3785Xn50xJvJAT03kbB3N77qIuzIWxGBGZtbGRhAsmrIOqOTs5wn3cPMHTVucbBiwWP9RwqlZ/CFxYKD66Q48p2dK 6B0+m1dx 2PWerxCAzizTXaYii7YcqCiIZAuBFFu12UK6Ts8+ydanR6g8iIowEgIwDSWpNQvEytgWQKmIP3VxFrF5ws9JLoGrnW9aXmI3CKH7yti+U4qRgo0E7AQlgiZFzCsu7P9MkKl1ImU1baYy+qt5Zeq0BSqjlzT/XmgFTaBTo9qSa6lnDJ6ruz64x9fVydgKNlyJpkKSJb/VMI/rc1Oiw+VNwxe5JCAMhmV5meAfRNVsdQnV7rpFJoeAQgOT2JhFXltfG8NK6TdEtXTt09AUidM3tA6GePKCOQcgTr/4Y2RQd0WNOcBWwjx1ZRcyJR4LNTYeOLzynEdu8pSNRhlKhAzG8guPhd8NN1+XOxB3I3W3UsgBb9ZablBIeLV0KXg5qRCyjyTvKBlzhbrc2OfIEiT+O4EyOJ3afOXBYe3WB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000007, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To keep vma locking correctness when vm_flags are modified, add modifier functions to be used whenever flags are updated. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 38 ++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 8 +++++++- 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ec2c4c227d51..35cf0a6cbcc2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -702,6 +702,44 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma_init_lock(vma); } +/* Use when VMA is not part of the VMA tree and needs no locking */ +static inline +void init_vm_flags(struct vm_area_struct *vma, unsigned long flags) +{ + WRITE_ONCE(vma->vm_flags, flags); +} + +/* Use when VMA is part of the VMA tree and needs appropriate locking */ +static inline +void reset_vm_flags(struct vm_area_struct *vma, unsigned long flags) +{ + vma_write_lock(vma); + init_vm_flags(vma, flags); +} + +static inline +void set_vm_flags(struct vm_area_struct *vma, unsigned long flags) +{ + vma_write_lock(vma); + vma->vm_flags |= flags; +} + +static inline +void clear_vm_flags(struct vm_area_struct *vma, unsigned long flags) +{ + vma_write_lock(vma); + vma->vm_flags &= ~flags; +} + +static inline +void mod_vm_flags(struct vm_area_struct *vma, + unsigned long set, unsigned long clear) +{ + vma_write_lock(vma); + vma->vm_flags |= set; + vma->vm_flags &= ~clear; +} + static inline void vma_set_anonymous(struct vm_area_struct *vma) { vma->vm_ops = NULL; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5f7c5ca89931..0d27edd3e63a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -553,7 +553,13 @@ struct vm_area_struct { * See vmf_insert_mixed_prot() for discussion. */ pgprot_t vm_page_prot; - unsigned long vm_flags; /* Flags, see mm.h. */ + + /* + * Flags, see mm.h. + * WARNING! Do not modify directly to keep correct VMA locking. + * Use {init|reset|set|clear|mod}_vm_flags() functions instead. + */ + unsigned long vm_flags; #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; From patchwork Mon Jan 9 20:53:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094280 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82EDEC6379F for ; Mon, 9 Jan 2023 20:54:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F79C8E0012; Mon, 9 Jan 2023 15:54:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A8978E0001; Mon, 9 Jan 2023 15:54:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3D5F8E0012; Mon, 9 Jan 2023 15:54:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E48D18E0001 for ; Mon, 9 Jan 2023 15:54:18 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B44F3A0BEE for ; Mon, 9 Jan 2023 20:54:18 +0000 (UTC) X-FDA: 80336463396.12.3AF75F9 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf08.hostedemail.com (Postfix) with ESMTP id 263B116000A for ; Mon, 9 Jan 2023 20:54:16 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jGV2twit; spf=pass (imf08.hostedemail.com: domain of 3-H68YwYKCA0574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3-H68YwYKCA0574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297657; a=rsa-sha256; cv=none; b=Tg/GlsqGM1RMe8A+5x/wFyX6rVea3J4TmAi+yzn5sI4e+r/Elp/6zo9L7Cr+n6Iw6CaxCF ZuWG2YSu+GVw2mZ2g/fQgj0v9iXikVCEwescsBptro3z2sxXW/BPRXGcwyB3d74cuqvwpq 0DJMqCR1zpGl+JZ9cCTja5Gyszjd0Po= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=jGV2twit; spf=pass (imf08.hostedemail.com: domain of 3-H68YwYKCA0574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3-H68YwYKCA0574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297657; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q5vG3v1PIcAm/ERdB03D9/j7bAwMWdEUWbQM9OP8Y4c=; b=lWHDJe5tU3/x4qWrYmHm3S2EpbTt+xttAifWOThFq7RJTHcjWtkIFDP4wiMP8soe3+PZMy XWiP2/XBMyKU+KMIzBx3eL7iRn7/9l924Gx7G8SGCPAs33GNbr0QHf/uHyBGf0u7FBZsvc IFd1D4ieT6NTTrTjHoBfrucmhSNQnxU= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-4c1456d608cso104264487b3.15 for ; Mon, 09 Jan 2023 12:54:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=q5vG3v1PIcAm/ERdB03D9/j7bAwMWdEUWbQM9OP8Y4c=; b=jGV2twit8eMqqlAwxSI2kIyw+GdbQsmn963oqlT3GN/ehTsmuPHj865pM8TWLhKQHd jTIMTXUStd0tYuBt7C6hb5YIOb4/wBppGWfSzDrzDKFeyuMa9uP2aNs8o441PZXFnjCo UTpNeX3UnEavX2v58G4+rqPOBU9aUnr6H/ChTcAvQIipWCjxL9GlTdM7oqLGXY19DPFQ q7bt4dBbBloqbwmYh9ylKkKgLjVpvssPEYXAiDYbhfBukg85oeSOt+1VHZNlGA7zFiYp 8fJla0RzQ4AZwln2gfdXjRilgUN42dHF3aEjUxK8uZvM/sfKPj53v6EzOYOtMmGPRMLa fuFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=q5vG3v1PIcAm/ERdB03D9/j7bAwMWdEUWbQM9OP8Y4c=; b=1SfxemTljAQrYVu5jQ1j+CLcOuFx1cejWtAbltLmJZXCcPnNmFvC1R7AzVtnoI1Hro kAlIN/Pd+ctJSoPcBQCIuCA5CCtGrMWFhhGOYf783zoKjRjfR8jrskDiRRviPR6eBdIl HYibA6xrTYDXhDiEKlpPCyI5hI5eKKbK2Zx/aOoJw7BSNcIVCtQmgzty/f70H6yFDnjF X0j+XDhcPDw9yaMQ3irwRvFEyVuCvHfKyL06yKuYFbwXrZaxNvacbVC9yE+t/N1OJ1RY lo0/fRvugpD+1pvd3yVrb1SteXolVwCaew9QWxKOrr+136KrcSVozCJtZatzD+MpcYlE sJjw== X-Gm-Message-State: AFqh2koFHiRUCr6FFc/ZEyLPY4XE5TtZHztkGYYRhaw7lmkbUSF8E+P5 tZo+hjNS89Q2nr34VjNJPGWNo1nmIsw= X-Google-Smtp-Source: AMrXdXvsQzJo8PSoW1ccRcrlqwtefe5KwIQ7Te6dX0P6/V4WPjQWsck4Zjq+DJXzzvFNYEa755Hhn8p6kg0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:ca88:0:b0:7bf:24c7:997e with SMTP id a130-20020a25ca88000000b007bf24c7997emr393826ybg.217.1673297656291; Mon, 09 Jan 2023 12:54:16 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:09 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-15-surenb@google.com> Subject: [PATCH 14/41] mm: replace VM_LOCKED_CLEAR_MASK with VM_LOCKED_MASK From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 263B116000A X-Rspamd-Server: rspam01 X-Stat-Signature: k7rodaj1waezs8fb9qn1w6rcjjmnir1w X-HE-Tag: 1673297656-296712 X-HE-Meta: U2FsdGVkX18SBvG3n45a67CAPaXxH3ssj/OqjA8NBcvi4N2VqZ9sYoG9VXdJkonQLctGPZjovc+RL7oQDV17Ysoq+xJ393s8KdOI3qj6t1SYq5mOVZJ28XiChtTsfzGmzZL/G+qCP205YHpZ8KzaHDKwuCQrMhizJW0Tyl+w4ty4XLpjWA6k4DEmhOqMLqFU2mgUlP3PSbau5zjc7YR+qiW9q462e90PbSdvGcCMk3z0o7kvHh7vsp6qnUpGOxX9FBkqNMCX1T79HiT47WQfUb3tHWdg82vr+ItZLqQ4y/chxPcTyjSKydnJDyTdQsZY/bEO6bFYstupfazL3WJu/iNq7HU4PS4wkY/YeI4SAYNHevqrRPTU0IViA7LDTFhnKgSu3y87hdhiD4jIcyzcGTk6tiZ9cxdKIjNh98kOYSpJLtU21f7qAyyAt2B4c7g4G7MyZnQ8DFSNyqulNVCFmSqWVvIlUt08uEkOwM8gLSBWl9m0qLOxPW2qN0dgoYWTuvfGIISxS2CDl8B6/Zt0Gh/hvzwbazv0rwR6Y2WUwrzEmC+J0PyfYcK/oUVPc9PKLUwbjPD4OOD0GbxTos30sWuUVCxWWoBGgxbNv09mjCU9aOHyyyC9WDYW9/BAfIfxKYCGTixH/Qwhhahl+hL5BaHPMpwhw8wcxT+yzXiYc8lkBgyQNsLAbk9Wg+F/m0C8c244xm2G7+peGPvZW2rnf4F9n3jdF3uKbZ5PXockMb9XeJG12maQSgG+aBx1iy1VJEjvKyRDqvD1Jck0sVFBxzGW3oXkTs389qEbGw/DPYBsYhWA6I4eh25KlfvZG3BzvDg7pn41XqbZky9T/VmA4NSGwilZayfLzpRk6YMs8Ol1v4LaXQe/kiOnBtgMqjHHhz8Yx14olLnUS4xqfZeiMSHB2nV9EGgLfPjIzLQusAwB02jTp6g3lqAkmmQI0y8r75AHLIkBfpdKACUuDWl 8Yub59p7 UTQ8guHqqL2ncfdbaXiecgBMej/ZwTfeFOpKjF/CvxfNFKW4MQAM/hmrLbHJ9aKAmGu2BuPIadVKO52fmFLqIrNwbpGXVngCFCpRsXNHwxYd+Zwptlut0AoXh0QhzcO+K1E6JKv1cIg4B6HpxbUBAp0lTexHhJ1TSbQar63tCuCLSLgjBz0jMGJi3IMqUcbEHSXM683uv2nLatpehAAs2xmYW2cG8adxh+nDS4R0mP3mtee3DrRUKBg8sYYNmozm1UByDborAHUQXbvu7bQF4SDyz+OjAL1GFu93UsOPttq0f7DRx+obuKMWdo0rAY/4tU5NJjSKDZ9oD39dog/vjzjRAMWALw5fr8wPTI0GSZ1z3jPpcKfyxNCBRfGsjbcAQpVM7H1VA3sDhg7+W/ElkyowA6soy5M0G5iBj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: To simplify the usage of VM_LOCKED_CLEAR_MASK in clear_vm_flags(), replace it with VM_LOCKED_MASK bitmask and convert all users. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 4 ++-- kernel/fork.c | 2 +- mm/hugetlb.c | 4 ++-- mm/mlock.c | 6 +++--- mm/mmap.c | 6 +++--- mm/mremap.c | 2 +- 6 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 35cf0a6cbcc2..2b16d45b75a6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -416,8 +416,8 @@ extern unsigned int kobjsize(const void *objp); /* This mask defines which mm->def_flags a process can inherit its parent */ #define VM_INIT_DEF_MASK VM_NOHUGEPAGE -/* This mask is used to clear all the VMA flags used by mlock */ -#define VM_LOCKED_CLEAR_MASK (~(VM_LOCKED | VM_LOCKONFAULT)) +/* This mask represents all the VMA flag bits used by mlock */ +#define VM_LOCKED_MASK (VM_LOCKED | VM_LOCKONFAULT) /* Arch-specific flags to clear when updating VM flags on protection change */ #ifndef VM_ARCH_CLEAR diff --git a/kernel/fork.c b/kernel/fork.c index c026d75108b3..1591dd8a0745 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -674,7 +674,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm, tmp->anon_vma = NULL; } else if (anon_vma_fork(tmp, mpnt)) goto fail_nomem_anon_vma_fork; - tmp->vm_flags &= ~(VM_LOCKED | VM_LOCKONFAULT); + clear_vm_flags(tmp, VM_LOCKED_MASK); file = tmp->vm_file; if (file) { struct address_space *mapping = file->f_mapping; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index db895230ee7e..24861cbfa2b1 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6950,8 +6950,8 @@ static unsigned long page_table_shareable(struct vm_area_struct *svma, unsigned long s_end = sbase + PUD_SIZE; /* Allow segments to share if only one is marked locked */ - unsigned long vm_flags = vma->vm_flags & VM_LOCKED_CLEAR_MASK; - unsigned long svm_flags = svma->vm_flags & VM_LOCKED_CLEAR_MASK; + unsigned long vm_flags = vma->vm_flags & ~VM_LOCKED_MASK; + unsigned long svm_flags = svma->vm_flags & ~VM_LOCKED_MASK; /* * match the virtual addresses, permission and the alignment of the diff --git a/mm/mlock.c b/mm/mlock.c index 7032f6dd0ce1..06aa9e204fac 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -490,7 +490,7 @@ static int apply_vma_lock_flags(unsigned long start, size_t len, prev = mas_prev(&mas, 0); for (nstart = start ; ; ) { - vm_flags_t newflags = vma->vm_flags & VM_LOCKED_CLEAR_MASK; + vm_flags_t newflags = vma->vm_flags & ~VM_LOCKED_MASK; newflags |= flags; @@ -662,7 +662,7 @@ static int apply_mlockall_flags(int flags) struct vm_area_struct *vma, *prev = NULL; vm_flags_t to_add = 0; - current->mm->def_flags &= VM_LOCKED_CLEAR_MASK; + current->mm->def_flags &= ~VM_LOCKED_MASK; if (flags & MCL_FUTURE) { current->mm->def_flags |= VM_LOCKED; @@ -682,7 +682,7 @@ static int apply_mlockall_flags(int flags) mas_for_each(&mas, vma, ULONG_MAX) { vm_flags_t newflags; - newflags = vma->vm_flags & VM_LOCKED_CLEAR_MASK; + newflags = vma->vm_flags & ~VM_LOCKED_MASK; newflags |= to_add; /* Ignore errors */ diff --git a/mm/mmap.c b/mm/mmap.c index 9db37adfc00a..5c4b608edde9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2721,7 +2721,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, if ((vm_flags & VM_SPECIAL) || vma_is_dax(vma) || is_vm_hugetlb_page(vma) || vma == get_gate_vma(current->mm)) - vma->vm_flags &= VM_LOCKED_CLEAR_MASK; + clear_vm_flags(vma, VM_LOCKED_MASK); else mm->locked_vm += (len >> PAGE_SHIFT); } @@ -3392,8 +3392,8 @@ static struct vm_area_struct *__install_special_mapping( vma->vm_start = addr; vma->vm_end = addr + len; - vma->vm_flags = vm_flags | mm->def_flags | VM_DONTEXPAND | VM_SOFTDIRTY; - vma->vm_flags &= VM_LOCKED_CLEAR_MASK; + init_vm_flags(vma, (vm_flags | mm->def_flags | + VM_DONTEXPAND | VM_SOFTDIRTY) & ~VM_LOCKED_MASK); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); vma->vm_ops = ops; diff --git a/mm/mremap.c b/mm/mremap.c index fe587c5d6591..5f6f9931bff1 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -686,7 +686,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, if (unlikely(!err && (flags & MREMAP_DONTUNMAP))) { /* We always clear VM_LOCKED[ONFAULT] on the old vma */ - vma->vm_flags &= VM_LOCKED_CLEAR_MASK; + clear_vm_flags(vma, VM_LOCKED_MASK); /* * anon_vma links of the old vma is no longer needed after its page From patchwork Mon Jan 9 20:53:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094282 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 149CAC5479D for ; Mon, 9 Jan 2023 20:54:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A82BA8E0013; Mon, 9 Jan 2023 15:54:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A59688E0001; Mon, 9 Jan 2023 15:54:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 85E948E0013; Mon, 9 Jan 2023 15:54:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6EA5C8E0001 for ; Mon, 9 Jan 2023 15:54:21 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4B2BDAB125 for ; Mon, 9 Jan 2023 20:54:21 +0000 (UTC) X-FDA: 80336463522.24.E7AEF8B Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf08.hostedemail.com (Postfix) with ESMTP id 7DBA9160011 for ; Mon, 9 Jan 2023 20:54:19 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=U85xBLsL; spf=pass (imf08.hostedemail.com: domain of 3-n68YwYKCA8796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-n68YwYKCA8796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297659; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6Ardk86ZWr85sWOJatKIPgo5mLXp8YCAbjCFaUECHgw=; b=r33t4LGaFl63A1Ta1Vre46AZ2puPPVG5zBWfGmXrEaWmNGK3D1dD4jbJqlAReDnA9mkpF8 xC+6KnzOCPJA4172CbjeN6gJaJlKsLy0q1QzNFpaL5DwQAHqkNQzAQAVvgyI6J/rwMSr4K vUiAEYQd0XIv08xLJYPcLitX4b/JhkI= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=U85xBLsL; spf=pass (imf08.hostedemail.com: domain of 3-n68YwYKCA8796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3-n68YwYKCA8796t2qv33v0t.r310x29C-11zAprz.36v@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297659; a=rsa-sha256; cv=none; b=zhAJjmBScNcVEyG5p7VHw10GwSKHiPjm1JHgdGl4E6PZ7rb9+LB84aMwiEEgwveoTrVzKx kctdIC8yXCuepra2OUFzJe8flse9U1YGnbPrP4PneoBjcunKi08D5TXk/3SwudKlHqlWb0 RuuwEvDRc3Wuk7Z/QbOi6+8i9ClPnlI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-46839d9ca5dso104894337b3.16 for ; Mon, 09 Jan 2023 12:54:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6Ardk86ZWr85sWOJatKIPgo5mLXp8YCAbjCFaUECHgw=; b=U85xBLsLgZCrqvXfEAg+BaZGz/oCE54YcPmN31yIJksHoum3GUFpvX6o/xRYHrKVyd E4a8yls32cSVlFzqD8P875zWgC/tJSi2iWYjbH9dUZozVCd+b4WdQPTCi9gGl7+7zj7A bQSa+/lZgMkuENk9nuHI7gV3Bhxr7XKQMhvJd8f0UiXEXL2tce0fkIW+hBTEhRWZnOgt /2tL4nNxLnQgCAC2A3SpLPpA792wJNiga2ra8JsoQUeBZO8Q3z22sDcpkTZ9PKzrpIa/ lk8fMvausMKZA9FfvsgHWY3VFM3qx2b48rKCPwhE5oifYwxJxANcNQsndjFCM9F7eiVG iPKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6Ardk86ZWr85sWOJatKIPgo5mLXp8YCAbjCFaUECHgw=; b=APvIPcwNjYl3h6ZJeaKqKKl6kl96ZIf3BLPvVfopQ8anowTM50xF003+ru2OK60klr VJSLOf70KMQeWSlouP2wa/suqX1QGmMbRtoimccHIdhlb1o24XlbWubGV4vQyHwcNga6 pwtOS+DUN4Ucq7PMElXTzrnJKFm0jGzKOVPgcRS7C3mKHoAlHrwKjEWDpWM6w+KYNXAL 14VkI1L9GiS8nUJh1MTzU24nxHcygtmG3KnD0Uy7bClnM7EYYy6kaU4PnYOXWQmwllPn XXuTRmu/FFzr42lf0rXJMo6tiB0/EbEDiAvmyOcNGB65K2d3kBORdv66gOD+J0VLzvSd U+Dw== X-Gm-Message-State: AFqh2koXCpRhEFXZecemr5pyVQ1xbhqcHPXhkODBF0m5YiOY3d0KAkGO kCELEzsusWGJHGbRIz8yO6dpA8wBMiI= X-Google-Smtp-Source: AMrXdXtHh3sXAXvU3ipTAaTFjEARVHgNgaZ/Vo1lplSDPhEcoTa76U/QAOd7/9m4ZYA8KiqxLFoWIR57kX0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:ef4b:0:b0:7b9:3895:a1e6 with SMTP id w11-20020a25ef4b000000b007b93895a1e6mr1365847ybm.335.1673297658938; Mon, 09 Jan 2023 12:54:18 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:10 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-16-surenb@google.com> Subject: [PATCH 15/41] mm: replace vma->vm_flags direct modifications with modifier calls From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: fbktj94wu9hzi5rb5o3qy8mur6z8gg8p X-Rspam-User: X-Rspamd-Queue-Id: 7DBA9160011 X-Rspamd-Server: rspam06 X-HE-Tag: 1673297659-845834 X-HE-Meta: U2FsdGVkX1+YIcZZTFvgTTqR7SS59CUYrhhEqbrhBOoTzdO9f081HER00aFS52iS7xnl/EtIPL9cfosv5OP02wpu0pWdtvDP7PuQaMUmKL1Znqgeom5Bd16whztpQgr0fOTXyRqm9bdFzXOq0fl3jX1INF7GxQZLgU7IWFQbt/V+KSzDm4GVUAUToT/eyQK39cBm9jCcPbtSCBflWvfF+7hFZ3dJWWXgxyL3O3WhAkf3zjV9B3ktsT1P9aSKCe5B/UL6ytvZXLGZxpXKKj5x4jMh5QT3fNR+dMS5FOOV4r90vMTFStSqUzbX1OBnRYFTiqm/2N/NkcekPV0TyiStF3mU+2iwukKQ2K2r+PhGHHPN/Z7cTrBvkTqPYQHxu4W+znZnPbNnIUmRfUPXVwTSdSs2DFJY7NZUNSDeEKeT60WORe/I5KizlysFnNduaLNKcCRUI61uGBVTTdxsOW1kNPIe2K1DigFl0Mml1gMWdg3Wrj7v9xraFLr15dpnMymk/b42Aom9QkYiSzWYFWnYIBIVFCZ972SGGgf2j1ao+5W9MvXOe28HL6K6pcoQDkayZUx0k7WNJ/zf1eSGF2n2ugpKKZld1A+WvqgK3OV4T7ZyZc1daJRpMrsnYnMm9U9k0iHhIXZdBT3c+8Fp6cgQV62PhiK7AwkhtrJV3+1v7fG9IRNFX61dGJJHFnRIkULGz4qosbQGbw2LaP4YE9cuHD4kRWfSthkFYO+3iQihFCR4sppb1MkTCFgUWG9Nr57TFXk1xFWq7tRmazqY9fqLkHLfz04SQQWyzskOpu1tQmp+pQyqbpUl8FWYnAlovn+P4F2Nx2owefRjvWIOu57nEKKaAdzflgddu1AvsX0jwgoJMzYZXpTDIs09pD4K3QXXHereZNmrzXeCx+OrGWXFOghUMl72+AmZ5ERDYaF4VkzsdD6yoXU+LOj9yNRsVmHpm9P4bHB2fSKmOaTwgaz hXk5wFU5 UzAyhG3D6MPX5VaZTMevg3S24uccmT4ECIfKP2e3l5gGYZmOxXNoVrxsRz4KjO2IexFjrab2dPAu2tkEho6ed64afQiCJlslz8hpULAcWbecxz0TWWw0jauqOfqfHH2VxSNdeEiGVAsd0A27skc5gpCQ2X2xWf+phvJ3jiSmtwKPrRVwDX41OWYxl4EUQtcK9+4XbOy5A3cEbDOpcTu/Uvp8Ilvr83lZhwkGoWoBdqMQBqhHHdKjMuThlbFE+Nncpow/CWVOYAMeVwxU/2OXOBpPcGAtvV5Jfus68TAQfPDSviJQakztwQJvLqmzta75+lVwOLgs52IubMahiuoJ0qHGMhGoNgdRuVcevWCfwb2Z443UTPwEGAn/Z6Nq7R/LUHaUMqidLNM3Y+3Fu2Bm3Q2yzVGghXF3lSd5J X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace direct modifications to vma->vm_flags with calls to modifier functions to be able to track flag changes and to keep vma locking correctness. Signed-off-by: Suren Baghdasaryan --- arch/arm/kernel/process.c | 2 +- arch/ia64/mm/init.c | 8 ++++---- arch/loongarch/include/asm/tlb.h | 2 +- arch/powerpc/kvm/book3s_xive_native.c | 2 +- arch/powerpc/mm/book3s64/subpage_prot.c | 2 +- arch/powerpc/platforms/book3s/vas-api.c | 2 +- arch/powerpc/platforms/cell/spufs/file.c | 14 +++++++------- arch/s390/mm/gmap.c | 3 +-- arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/kernel/cpu/sgx/driver.c | 2 +- arch/x86/kernel/cpu/sgx/virt.c | 2 +- arch/x86/mm/pat/memtype.c | 6 +++--- arch/x86/um/mem_32.c | 2 +- drivers/acpi/pfr_telemetry.c | 2 +- drivers/android/binder.c | 3 +-- drivers/char/mspec.c | 2 +- drivers/crypto/hisilicon/qm.c | 2 +- drivers/dax/device.c | 2 +- drivers/dma/idxd/cdev.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 2 +- drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 4 ++-- drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c | 4 ++-- drivers/gpu/drm/amd/amdkfd/kfd_events.c | 4 ++-- drivers/gpu/drm/amd/amdkfd/kfd_process.c | 4 ++-- drivers/gpu/drm/drm_gem.c | 2 +- drivers/gpu/drm/drm_gem_dma_helper.c | 3 +-- drivers/gpu/drm/drm_gem_shmem_helper.c | 2 +- drivers/gpu/drm/drm_vm.c | 8 ++++---- drivers/gpu/drm/etnaviv/etnaviv_gem.c | 2 +- drivers/gpu/drm/exynos/exynos_drm_gem.c | 4 ++-- drivers/gpu/drm/gma500/framebuffer.c | 2 +- drivers/gpu/drm/i810/i810_dma.c | 2 +- drivers/gpu/drm/i915/gem/i915_gem_mman.c | 4 ++-- drivers/gpu/drm/mediatek/mtk_drm_gem.c | 2 +- drivers/gpu/drm/msm/msm_gem.c | 2 +- drivers/gpu/drm/omapdrm/omap_gem.c | 3 +-- drivers/gpu/drm/rockchip/rockchip_drm_gem.c | 3 +-- drivers/gpu/drm/tegra/gem.c | 5 ++--- drivers/gpu/drm/ttm/ttm_bo_vm.c | 3 +-- drivers/gpu/drm/virtio/virtgpu_vram.c | 2 +- drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c | 2 +- drivers/gpu/drm/xen/xen_drm_front_gem.c | 3 +-- drivers/hsi/clients/cmt_speech.c | 2 +- drivers/hwtracing/intel_th/msu.c | 2 +- drivers/hwtracing/stm/core.c | 2 +- drivers/infiniband/hw/hfi1/file_ops.c | 4 ++-- drivers/infiniband/hw/mlx5/main.c | 4 ++-- drivers/infiniband/hw/qib/qib_file_ops.c | 13 ++++++------- drivers/infiniband/hw/usnic/usnic_ib_verbs.c | 2 +- drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c | 2 +- .../media/common/videobuf2/videobuf2-dma-contig.c | 2 +- drivers/media/common/videobuf2/videobuf2-vmalloc.c | 2 +- drivers/media/v4l2-core/videobuf-dma-contig.c | 2 +- drivers/media/v4l2-core/videobuf-dma-sg.c | 4 ++-- drivers/media/v4l2-core/videobuf-vmalloc.c | 2 +- drivers/misc/cxl/context.c | 2 +- drivers/misc/habanalabs/common/memory.c | 2 +- drivers/misc/habanalabs/gaudi/gaudi.c | 4 ++-- drivers/misc/habanalabs/gaudi2/gaudi2.c | 8 ++++---- drivers/misc/habanalabs/goya/goya.c | 4 ++-- drivers/misc/ocxl/context.c | 4 ++-- drivers/misc/ocxl/sysfs.c | 2 +- drivers/misc/open-dice.c | 6 +++--- drivers/misc/sgi-gru/grufile.c | 4 ++-- drivers/misc/uacce/uacce.c | 2 +- drivers/sbus/char/oradax.c | 2 +- drivers/scsi/cxlflash/ocxl_hw.c | 2 +- drivers/scsi/sg.c | 2 +- drivers/staging/media/atomisp/pci/hmm/hmm_bo.c | 2 +- drivers/staging/media/deprecated/meye/meye.c | 4 ++-- .../media/deprecated/stkwebcam/stk-webcam.c | 2 +- drivers/target/target_core_user.c | 2 +- drivers/uio/uio.c | 2 +- drivers/usb/core/devio.c | 3 +-- drivers/usb/mon/mon_bin.c | 3 +-- drivers/vdpa/vdpa_user/iova_domain.c | 2 +- drivers/vfio/pci/vfio_pci_core.c | 2 +- drivers/vhost/vdpa.c | 2 +- drivers/video/fbdev/68328fb.c | 2 +- drivers/video/fbdev/core/fb_defio.c | 4 ++-- drivers/xen/gntalloc.c | 2 +- drivers/xen/gntdev.c | 4 ++-- drivers/xen/privcmd-buf.c | 2 +- drivers/xen/privcmd.c | 4 ++-- fs/aio.c | 2 +- fs/cramfs/inode.c | 2 +- fs/erofs/data.c | 2 +- fs/exec.c | 4 ++-- fs/ext4/file.c | 2 +- fs/fuse/dax.c | 2 +- fs/hugetlbfs/inode.c | 4 ++-- fs/orangefs/file.c | 3 +-- fs/proc/task_mmu.c | 2 +- fs/proc/vmcore.c | 3 +-- fs/userfaultfd.c | 12 ++++++------ fs/xfs/xfs_file.c | 2 +- include/linux/mm.h | 2 +- kernel/bpf/ringbuf.c | 4 ++-- kernel/bpf/syscall.c | 4 ++-- kernel/events/core.c | 2 +- kernel/kcov.c | 2 +- kernel/relay.c | 2 +- mm/madvise.c | 2 +- mm/memory.c | 6 +++--- mm/mlock.c | 6 +++--- mm/mmap.c | 10 +++++----- mm/mprotect.c | 2 +- mm/mremap.c | 6 +++--- mm/nommu.c | 11 ++++++----- mm/secretmem.c | 2 +- mm/shmem.c | 2 +- mm/vmalloc.c | 2 +- net/ipv4/tcp.c | 4 ++-- security/selinux/selinuxfs.c | 6 +++--- sound/core/oss/pcm_oss.c | 2 +- sound/core/pcm_native.c | 9 +++++---- sound/soc/pxa/mmp-sspa.c | 2 +- sound/usb/usx2y/us122l.c | 4 ++-- sound/usb/usx2y/usX2Yhwdep.c | 2 +- sound/usb/usx2y/usx2yhwdeppcm.c | 2 +- 120 files changed, 194 insertions(+), 205 deletions(-) diff --git a/arch/arm/kernel/process.c b/arch/arm/kernel/process.c index f811733a8fc5..ec65f3ea3150 100644 --- a/arch/arm/kernel/process.c +++ b/arch/arm/kernel/process.c @@ -316,7 +316,7 @@ static int __init gate_vma_init(void) gate_vma.vm_page_prot = PAGE_READONLY_EXEC; gate_vma.vm_start = 0xffff0000; gate_vma.vm_end = 0xffff0000 + PAGE_SIZE; - gate_vma.vm_flags = VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYEXEC; + init_vm_flags(&gate_vma, VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYEXEC); return 0; } arch_initcall(gate_vma_init); diff --git a/arch/ia64/mm/init.c b/arch/ia64/mm/init.c index fc4e4217e87f..d355e0ce28ab 100644 --- a/arch/ia64/mm/init.c +++ b/arch/ia64/mm/init.c @@ -109,7 +109,7 @@ ia64_init_addr_space (void) vma_set_anonymous(vma); vma->vm_start = current->thread.rbs_bot & PAGE_MASK; vma->vm_end = vma->vm_start + PAGE_SIZE; - vma->vm_flags = VM_DATA_DEFAULT_FLAGS|VM_GROWSUP|VM_ACCOUNT; + init_vm_flags(vma, VM_DATA_DEFAULT_FLAGS|VM_GROWSUP|VM_ACCOUNT); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); mmap_write_lock(current->mm); if (insert_vm_struct(current->mm, vma)) { @@ -127,8 +127,8 @@ ia64_init_addr_space (void) vma_set_anonymous(vma); vma->vm_end = PAGE_SIZE; vma->vm_page_prot = __pgprot(pgprot_val(PAGE_READONLY) | _PAGE_MA_NAT); - vma->vm_flags = VM_READ | VM_MAYREAD | VM_IO | - VM_DONTEXPAND | VM_DONTDUMP; + init_vm_flags(vma, VM_READ | VM_MAYREAD | VM_IO | + VM_DONTEXPAND | VM_DONTDUMP); mmap_write_lock(current->mm); if (insert_vm_struct(current->mm, vma)) { mmap_write_unlock(current->mm); @@ -272,7 +272,7 @@ static int __init gate_vma_init(void) vma_init(&gate_vma, NULL); gate_vma.vm_start = FIXADDR_USER_START; gate_vma.vm_end = FIXADDR_USER_END; - gate_vma.vm_flags = VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC; + init_vm_flags(&gate_vma, VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC); gate_vma.vm_page_prot = __pgprot(__ACCESS_BITS | _PAGE_PL_3 | _PAGE_AR_RX); return 0; diff --git a/arch/loongarch/include/asm/tlb.h b/arch/loongarch/include/asm/tlb.h index dd24f5898f65..51e35b44d105 100644 --- a/arch/loongarch/include/asm/tlb.h +++ b/arch/loongarch/include/asm/tlb.h @@ -149,7 +149,7 @@ static inline void tlb_flush(struct mmu_gather *tlb) struct vm_area_struct vma; vma.vm_mm = tlb->mm; - vma.vm_flags = 0; + init_vm_flags(&vma, 0); if (tlb->fullmm) { flush_tlb_mm(tlb->mm); return; diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c index 4f566bea5e10..7976af0f5ff8 100644 --- a/arch/powerpc/kvm/book3s_xive_native.c +++ b/arch/powerpc/kvm/book3s_xive_native.c @@ -324,7 +324,7 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev, return -EINVAL; } - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot); /* diff --git a/arch/powerpc/mm/book3s64/subpage_prot.c b/arch/powerpc/mm/book3s64/subpage_prot.c index d73b3b4176e8..72948cdb1911 100644 --- a/arch/powerpc/mm/book3s64/subpage_prot.c +++ b/arch/powerpc/mm/book3s64/subpage_prot.c @@ -156,7 +156,7 @@ static void subpage_mark_vma_nohuge(struct mm_struct *mm, unsigned long addr, * VM_NOHUGEPAGE and split them. */ for_each_vma_range(vmi, vma, addr + len) { - vma->vm_flags |= VM_NOHUGEPAGE; + set_vm_flags(vma, VM_NOHUGEPAGE); walk_page_vma(vma, &subpage_walk_ops, NULL); } } diff --git a/arch/powerpc/platforms/book3s/vas-api.c b/arch/powerpc/platforms/book3s/vas-api.c index eb5bed333750..a81615768fff 100644 --- a/arch/powerpc/platforms/book3s/vas-api.c +++ b/arch/powerpc/platforms/book3s/vas-api.c @@ -525,7 +525,7 @@ static int coproc_mmap(struct file *fp, struct vm_area_struct *vma) pfn = paste_addr >> PAGE_SHIFT; /* flags, page_prot from cxl_mmap(), except we want cachable */ - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_cached(vma->vm_page_prot); prot = __pgprot(pgprot_val(vma->vm_page_prot) | _PAGE_DIRTY); diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c index 62d90a5e23d1..784fa39a484a 100644 --- a/arch/powerpc/platforms/cell/spufs/file.c +++ b/arch/powerpc/platforms/cell/spufs/file.c @@ -291,7 +291,7 @@ static int spufs_mem_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot); vma->vm_ops = &spufs_mem_mmap_vmops; @@ -381,7 +381,7 @@ static int spufs_cntl_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_cntl_mmap_vmops; @@ -1043,7 +1043,7 @@ static int spufs_signal1_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_signal1_mmap_vmops; @@ -1179,7 +1179,7 @@ static int spufs_signal2_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_signal2_mmap_vmops; @@ -1302,7 +1302,7 @@ static int spufs_mss_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_mss_mmap_vmops; @@ -1364,7 +1364,7 @@ static int spufs_psmap_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_psmap_mmap_vmops; @@ -1424,7 +1424,7 @@ static int spufs_mfc_mmap(struct file *file, struct vm_area_struct *vma) if (!(vma->vm_flags & VM_SHARED)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &spufs_mfc_mmap_vmops; diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 74e1d873dce0..3811d6c86d09 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2522,8 +2522,7 @@ static inline void thp_split_mm(struct mm_struct *mm) VMA_ITERATOR(vmi, mm, 0); for_each_vma(vmi, vma) { - vma->vm_flags &= ~VM_HUGEPAGE; - vma->vm_flags |= VM_NOHUGEPAGE; + mod_vm_flags(vma, VM_NOHUGEPAGE, VM_HUGEPAGE); walk_page_vma(vma, &thp_split_walk_ops, NULL); } mm->def_flags |= VM_NOHUGEPAGE; diff --git a/arch/x86/entry/vsyscall/vsyscall_64.c b/arch/x86/entry/vsyscall/vsyscall_64.c index 4af81df133ee..e2a1626d86d8 100644 --- a/arch/x86/entry/vsyscall/vsyscall_64.c +++ b/arch/x86/entry/vsyscall/vsyscall_64.c @@ -391,7 +391,7 @@ void __init map_vsyscall(void) } if (vsyscall_mode == XONLY) - gate_vma.vm_flags = VM_EXEC; + init_vm_flags(&gate_vma, VM_EXEC); BUILD_BUG_ON((unsigned long)__fix_to_virt(VSYSCALL_PAGE) != (unsigned long)VSYSCALL_ADDR); diff --git a/arch/x86/kernel/cpu/sgx/driver.c b/arch/x86/kernel/cpu/sgx/driver.c index aa9b8b868867..42c0bded93b6 100644 --- a/arch/x86/kernel/cpu/sgx/driver.c +++ b/arch/x86/kernel/cpu/sgx/driver.c @@ -95,7 +95,7 @@ static int sgx_mmap(struct file *file, struct vm_area_struct *vma) return ret; vma->vm_ops = &sgx_vm_ops; - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO); vma->vm_private_data = encl; return 0; diff --git a/arch/x86/kernel/cpu/sgx/virt.c b/arch/x86/kernel/cpu/sgx/virt.c index 6a77a14eee38..0774a0bfeb28 100644 --- a/arch/x86/kernel/cpu/sgx/virt.c +++ b/arch/x86/kernel/cpu/sgx/virt.c @@ -105,7 +105,7 @@ static int sgx_vepc_mmap(struct file *file, struct vm_area_struct *vma) vma->vm_ops = &sgx_vepc_vm_ops; /* Don't copy VMA in fork() */ - vma->vm_flags |= VM_PFNMAP | VM_IO | VM_DONTDUMP | VM_DONTCOPY; + set_vm_flags(vma, VM_PFNMAP | VM_IO | VM_DONTDUMP | VM_DONTCOPY); vma->vm_private_data = vepc; return 0; diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index 46de9cf5c91d..9e490a372896 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -999,7 +999,7 @@ int track_pfn_remap(struct vm_area_struct *vma, pgprot_t *prot, ret = reserve_pfn_range(paddr, size, prot, 0); if (ret == 0 && vma) - vma->vm_flags |= VM_PAT; + set_vm_flags(vma, VM_PAT); return ret; } @@ -1065,7 +1065,7 @@ void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, } free_pfn_range(paddr, size); if (vma) - vma->vm_flags &= ~VM_PAT; + clear_vm_flags(vma, VM_PAT); } /* @@ -1075,7 +1075,7 @@ void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, */ void untrack_pfn_moved(struct vm_area_struct *vma) { - vma->vm_flags &= ~VM_PAT; + clear_vm_flags(vma, VM_PAT); } pgprot_t pgprot_writecombine(pgprot_t prot) diff --git a/arch/x86/um/mem_32.c b/arch/x86/um/mem_32.c index cafd01f730da..bfd2c320ad25 100644 --- a/arch/x86/um/mem_32.c +++ b/arch/x86/um/mem_32.c @@ -16,7 +16,7 @@ static int __init gate_vma_init(void) vma_init(&gate_vma, NULL); gate_vma.vm_start = FIXADDR_USER_START; gate_vma.vm_end = FIXADDR_USER_END; - gate_vma.vm_flags = VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC; + init_vm_flags(&gate_vma, VM_READ | VM_MAYREAD | VM_EXEC | VM_MAYEXEC); gate_vma.vm_page_prot = PAGE_READONLY; return 0; diff --git a/drivers/acpi/pfr_telemetry.c b/drivers/acpi/pfr_telemetry.c index 27fb6cdad75f..9e339c705b5b 100644 --- a/drivers/acpi/pfr_telemetry.c +++ b/drivers/acpi/pfr_telemetry.c @@ -310,7 +310,7 @@ pfrt_log_mmap(struct file *file, struct vm_area_struct *vma) return -EROFS; /* changing from read to write with mprotect is not allowed */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); pfrt_log_dev = to_pfrt_log_dev(file); diff --git a/drivers/android/binder.c b/drivers/android/binder.c index 880224ec6abb..dd6c99223b8c 100644 --- a/drivers/android/binder.c +++ b/drivers/android/binder.c @@ -5572,8 +5572,7 @@ static int binder_mmap(struct file *filp, struct vm_area_struct *vma) proc->pid, vma->vm_start, vma->vm_end, "bad vm_flags", -EPERM); return -EPERM; } - vma->vm_flags |= VM_DONTCOPY | VM_MIXEDMAP; - vma->vm_flags &= ~VM_MAYWRITE; + mod_vm_flags(vma, VM_DONTCOPY | VM_MIXEDMAP, VM_MAYWRITE); vma->vm_ops = &binder_vm_ops; vma->vm_private_data = proc; diff --git a/drivers/char/mspec.c b/drivers/char/mspec.c index f8231e2e84be..57bd36a28f95 100644 --- a/drivers/char/mspec.c +++ b/drivers/char/mspec.c @@ -206,7 +206,7 @@ mspec_mmap(struct file *file, struct vm_area_struct *vma, refcount_set(&vdata->refcnt, 1); vma->vm_private_data = vdata; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); if (vdata->type == MSPEC_UNCACHED) vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &mspec_vm_ops; diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c index 007ac7a69ce7..57ecdb5c97fb 100644 --- a/drivers/crypto/hisilicon/qm.c +++ b/drivers/crypto/hisilicon/qm.c @@ -2363,7 +2363,7 @@ static int hisi_qm_uacce_mmap(struct uacce_queue *q, return -EINVAL; } - vma->vm_flags |= VM_IO; + set_vm_flags(vma, VM_IO); return remap_pfn_range(vma, vma->vm_start, phys_base >> PAGE_SHIFT, diff --git a/drivers/dax/device.c b/drivers/dax/device.c index 5494d745ced5..6e9726dfaa7e 100644 --- a/drivers/dax/device.c +++ b/drivers/dax/device.c @@ -308,7 +308,7 @@ static int dax_mmap(struct file *filp, struct vm_area_struct *vma) return rc; vma->vm_ops = &dax_vm_ops; - vma->vm_flags |= VM_HUGEPAGE; + set_vm_flags(vma, VM_HUGEPAGE); return 0; } diff --git a/drivers/dma/idxd/cdev.c b/drivers/dma/idxd/cdev.c index e13e92609943..51cf836cf329 100644 --- a/drivers/dma/idxd/cdev.c +++ b/drivers/dma/idxd/cdev.c @@ -201,7 +201,7 @@ static int idxd_cdev_mmap(struct file *filp, struct vm_area_struct *vma) if (rc < 0) return rc; - vma->vm_flags |= VM_DONTCOPY; + set_vm_flags(vma, VM_DONTCOPY); pfn = (base + idxd_get_wq_portal_full_offset(wq->id, IDXD_PORTAL_LIMITED)) >> PAGE_SHIFT; vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c index bb7350ea1d75..70b08a0d13cd 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c @@ -257,7 +257,7 @@ static int amdgpu_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_str */ if (is_cow_mapping(vma->vm_flags) && !(vma->vm_flags & VM_ACCESS_FLAGS)) - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); return drm_gem_ttm_mmap(obj, vma); } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index 6d291aa6386b..7beb8dd6a5e6 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -2879,8 +2879,8 @@ static int kfd_mmio_mmap(struct kfd_dev *dev, struct kfd_process *process, address = dev->adev->rmmio_remap.bus_addr; - vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE | - VM_DONTDUMP | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE | + VM_DONTDUMP | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c index cd4e61bf0493..6cbe47cf9be5 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c @@ -159,8 +159,8 @@ int kfd_doorbell_mmap(struct kfd_dev *dev, struct kfd_process *process, address = kfd_get_process_doorbells(pdd); if (!address) return -ENOMEM; - vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE | - VM_DONTDUMP | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE | + VM_DONTDUMP | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_events.c index 729d26d648af..95cd20056cea 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_events.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_events.c @@ -1052,8 +1052,8 @@ int kfd_event_mmap(struct kfd_process *p, struct vm_area_struct *vma) pfn = __pa(page->kernel_address); pfn >>= PAGE_SHIFT; - vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE - | VM_DONTDUMP | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_DONTCOPY | VM_DONTEXPAND | VM_NORESERVE + | VM_DONTDUMP | VM_PFNMAP); pr_debug("Mapping signal page\n"); pr_debug(" start user address == 0x%08lx\n", vma->vm_start); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index 51b1683ac5c1..b40f4b122918 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -1978,8 +1978,8 @@ int kfd_reserved_mem_mmap(struct kfd_dev *dev, struct kfd_process *process, return -ENOMEM; } - vma->vm_flags |= VM_IO | VM_DONTCOPY | VM_DONTEXPAND - | VM_NORESERVE | VM_DONTDUMP | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_DONTCOPY | VM_DONTEXPAND + | VM_NORESERVE | VM_DONTDUMP | VM_PFNMAP); /* Mapping pages to user process */ return remap_pfn_range(vma, vma->vm_start, PFN_DOWN(__pa(qpd->cwsr_kaddr)), diff --git a/drivers/gpu/drm/drm_gem.c b/drivers/gpu/drm/drm_gem.c index b8db675e7fb5..6ea7bcaa592b 100644 --- a/drivers/gpu/drm/drm_gem.c +++ b/drivers/gpu/drm/drm_gem.c @@ -1047,7 +1047,7 @@ int drm_gem_mmap_obj(struct drm_gem_object *obj, unsigned long obj_size, goto err_drm_gem_object_put; } - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags)); vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); } diff --git a/drivers/gpu/drm/drm_gem_dma_helper.c b/drivers/gpu/drm/drm_gem_dma_helper.c index 1e658c448366..41f241b9a581 100644 --- a/drivers/gpu/drm/drm_gem_dma_helper.c +++ b/drivers/gpu/drm/drm_gem_dma_helper.c @@ -530,8 +530,7 @@ int drm_gem_dma_mmap(struct drm_gem_dma_object *dma_obj, struct vm_area_struct * * the whole buffer. */ vma->vm_pgoff -= drm_vma_node_start(&obj->vma_node); - vma->vm_flags &= ~VM_PFNMAP; - vma->vm_flags |= VM_DONTEXPAND; + mod_vm_flags(vma, VM_DONTEXPAND, VM_PFNMAP); if (dma_obj->map_noncoherent) { vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c b/drivers/gpu/drm/drm_gem_shmem_helper.c index b602cd72a120..a5032dfac492 100644 --- a/drivers/gpu/drm/drm_gem_shmem_helper.c +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c @@ -633,7 +633,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct if (ret) return ret; - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); if (shmem->map_wc) vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); diff --git a/drivers/gpu/drm/drm_vm.c b/drivers/gpu/drm/drm_vm.c index f024dc93939e..8867bb6c40e3 100644 --- a/drivers/gpu/drm/drm_vm.c +++ b/drivers/gpu/drm/drm_vm.c @@ -476,7 +476,7 @@ static int drm_mmap_dma(struct file *filp, struct vm_area_struct *vma) if (!capable(CAP_SYS_ADMIN) && (dma->flags & _DRM_DMA_USE_PCI_RO)) { - vma->vm_flags &= ~(VM_WRITE | VM_MAYWRITE); + clear_vm_flags(vma, VM_WRITE | VM_MAYWRITE); #if defined(__i386__) || defined(__x86_64__) pgprot_val(vma->vm_page_prot) &= ~_PAGE_RW; #else @@ -492,7 +492,7 @@ static int drm_mmap_dma(struct file *filp, struct vm_area_struct *vma) vma->vm_ops = &drm_vm_dma_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); drm_vm_open_locked(dev, vma); return 0; @@ -560,7 +560,7 @@ static int drm_mmap_locked(struct file *filp, struct vm_area_struct *vma) return -EINVAL; if (!capable(CAP_SYS_ADMIN) && (map->flags & _DRM_READ_ONLY)) { - vma->vm_flags &= ~(VM_WRITE | VM_MAYWRITE); + clear_vm_flags(vma, VM_WRITE | VM_MAYWRITE); #if defined(__i386__) || defined(__x86_64__) pgprot_val(vma->vm_page_prot) &= ~_PAGE_RW; #else @@ -628,7 +628,7 @@ static int drm_mmap_locked(struct file *filp, struct vm_area_struct *vma) default: return -EINVAL; /* This should never happen. */ } - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); drm_vm_open_locked(dev, vma); return 0; diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.c b/drivers/gpu/drm/etnaviv/etnaviv_gem.c index c5ae5492e1af..9a5a317038a4 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_gem.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.c @@ -130,7 +130,7 @@ static int etnaviv_gem_mmap_obj(struct etnaviv_gem_object *etnaviv_obj, { pgprot_t vm_page_prot; - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vm_page_prot = vm_get_page_prot(vma->vm_flags); diff --git a/drivers/gpu/drm/exynos/exynos_drm_gem.c b/drivers/gpu/drm/exynos/exynos_drm_gem.c index 3e493f48e0d4..c330d415729c 100644 --- a/drivers/gpu/drm/exynos/exynos_drm_gem.c +++ b/drivers/gpu/drm/exynos/exynos_drm_gem.c @@ -274,7 +274,7 @@ static int exynos_drm_gem_mmap_buffer(struct exynos_drm_gem *exynos_gem, unsigned long vm_size; int ret; - vma->vm_flags &= ~VM_PFNMAP; + clear_vm_flags(vma, VM_PFNMAP); vma->vm_pgoff = 0; vm_size = vma->vm_end - vma->vm_start; @@ -368,7 +368,7 @@ static int exynos_drm_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct if (obj->import_attach) return dma_buf_mmap(obj->dma_buf, vma, 0); - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); DRM_DEV_DEBUG_KMS(to_dma_dev(obj->dev), "flags = 0x%x\n", exynos_gem->flags); diff --git a/drivers/gpu/drm/gma500/framebuffer.c b/drivers/gpu/drm/gma500/framebuffer.c index 8d5a37b8f110..471d5b3c1535 100644 --- a/drivers/gpu/drm/gma500/framebuffer.c +++ b/drivers/gpu/drm/gma500/framebuffer.c @@ -139,7 +139,7 @@ static int psbfb_mmap(struct fb_info *info, struct vm_area_struct *vma) */ vma->vm_ops = &psbfb_vm_ops; vma->vm_private_data = (void *)fb; - vma->vm_flags |= VM_IO | VM_MIXEDMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_MIXEDMAP | VM_DONTEXPAND | VM_DONTDUMP); return 0; } diff --git a/drivers/gpu/drm/i810/i810_dma.c b/drivers/gpu/drm/i810/i810_dma.c index 9fb4dd63342f..bced8c30709e 100644 --- a/drivers/gpu/drm/i810/i810_dma.c +++ b/drivers/gpu/drm/i810/i810_dma.c @@ -102,7 +102,7 @@ static int i810_mmap_buffers(struct file *filp, struct vm_area_struct *vma) buf = dev_priv->mmap_buffer; buf_priv = buf->dev_private; - vma->vm_flags |= VM_DONTCOPY; + set_vm_flags(vma, VM_DONTCOPY); buf_priv->currently_mapped = I810_BUF_MAPPED; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/i915_gem_mman.c index 0ad44f3868de..71b9e0485cb9 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_mman.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_mman.c @@ -979,7 +979,7 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma) i915_gem_object_put(obj); return -EINVAL; } - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } anon = mmap_singleton(to_i915(dev)); @@ -988,7 +988,7 @@ int i915_gem_mmap(struct file *filp, struct vm_area_struct *vma) return PTR_ERR(anon); } - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | VM_IO); /* * We keep the ref on mmo->obj, not vm_file, but we require diff --git a/drivers/gpu/drm/mediatek/mtk_drm_gem.c b/drivers/gpu/drm/mediatek/mtk_drm_gem.c index 47e96b0289f9..427089733b87 100644 --- a/drivers/gpu/drm/mediatek/mtk_drm_gem.c +++ b/drivers/gpu/drm/mediatek/mtk_drm_gem.c @@ -158,7 +158,7 @@ static int mtk_drm_gem_object_mmap(struct drm_gem_object *obj, * dma_alloc_attrs() allocated a struct page table for mtk_gem, so clear * VM_PFNMAP flag that was set by drm_gem_mmap_obj()/drm_gem_mmap(). */ - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags)); vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); diff --git a/drivers/gpu/drm/msm/msm_gem.c b/drivers/gpu/drm/msm/msm_gem.c index 1dee0d18abbb..8aff3ae909af 100644 --- a/drivers/gpu/drm/msm/msm_gem.c +++ b/drivers/gpu/drm/msm/msm_gem.c @@ -1012,7 +1012,7 @@ static int msm_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_struct { struct msm_gem_object *msm_obj = to_msm_bo(obj); - vma->vm_flags |= VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = msm_gem_pgprot(msm_obj, vm_get_page_prot(vma->vm_flags)); return 0; diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c index cf571796fd26..9c0e7d6a3784 100644 --- a/drivers/gpu/drm/omapdrm/omap_gem.c +++ b/drivers/gpu/drm/omapdrm/omap_gem.c @@ -543,8 +543,7 @@ int omap_gem_mmap_obj(struct drm_gem_object *obj, { struct omap_gem_object *omap_obj = to_omap_bo(obj); - vma->vm_flags &= ~VM_PFNMAP; - vma->vm_flags |= VM_MIXEDMAP; + mod_vm_flags(vma, VM_MIXEDMAP, VM_PFNMAP); if (omap_obj->flags & OMAP_BO_WC) { vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags)); diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_gem.c b/drivers/gpu/drm/rockchip/rockchip_drm_gem.c index 6edb7c52cb3d..735b64bbdcf2 100644 --- a/drivers/gpu/drm/rockchip/rockchip_drm_gem.c +++ b/drivers/gpu/drm/rockchip/rockchip_drm_gem.c @@ -251,8 +251,7 @@ static int rockchip_drm_gem_object_mmap(struct drm_gem_object *obj, * We allocated a struct page table for rk_obj, so clear * VM_PFNMAP flag that was set by drm_gem_mmap_obj()/drm_gem_mmap(). */ - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; - vma->vm_flags &= ~VM_PFNMAP; + mod_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP, VM_PFNMAP); vma->vm_page_prot = pgprot_writecombine(vm_get_page_prot(vma->vm_flags)); vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c index 979e7bc902f6..6cdc6c45ef27 100644 --- a/drivers/gpu/drm/tegra/gem.c +++ b/drivers/gpu/drm/tegra/gem.c @@ -574,7 +574,7 @@ int __tegra_gem_mmap(struct drm_gem_object *gem, struct vm_area_struct *vma) * and set the vm_pgoff (used as a fake buffer offset by DRM) * to 0 as we want to map the whole buffer. */ - vma->vm_flags &= ~VM_PFNMAP; + clear_vm_flags(vma, VM_PFNMAP); vma->vm_pgoff = 0; err = dma_mmap_wc(gem->dev->dev, vma, bo->vaddr, bo->iova, @@ -588,8 +588,7 @@ int __tegra_gem_mmap(struct drm_gem_object *gem, struct vm_area_struct *vma) } else { pgprot_t prot = vm_get_page_prot(vma->vm_flags); - vma->vm_flags |= VM_MIXEDMAP; - vma->vm_flags &= ~VM_PFNMAP; + mod_vm_flags(vma, VM_MIXEDMAP, VM_PFNMAP); vma->vm_page_prot = pgprot_writecombine(prot); } diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c index 5a3e4b891377..0861e6e33964 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_vm.c +++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c @@ -468,8 +468,7 @@ int ttm_bo_mmap_obj(struct vm_area_struct *vma, struct ttm_buffer_object *bo) vma->vm_private_data = bo; - vma->vm_flags |= VM_PFNMAP; - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_PFNMAP | VM_IO | VM_DONTEXPAND | VM_DONTDUMP); return 0; } EXPORT_SYMBOL(ttm_bo_mmap_obj); diff --git a/drivers/gpu/drm/virtio/virtgpu_vram.c b/drivers/gpu/drm/virtio/virtgpu_vram.c index 6b45b0429fef..5498a1dbef63 100644 --- a/drivers/gpu/drm/virtio/virtgpu_vram.c +++ b/drivers/gpu/drm/virtio/virtgpu_vram.c @@ -46,7 +46,7 @@ static int virtio_gpu_vram_mmap(struct drm_gem_object *obj, return -EINVAL; vma->vm_pgoff -= drm_vma_node_start(&obj->vma_node); - vma->vm_flags |= VM_MIXEDMAP | VM_DONTEXPAND; + set_vm_flags(vma, VM_MIXEDMAP | VM_DONTEXPAND); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); vma->vm_page_prot = pgprot_decrypted(vma->vm_page_prot); vma->vm_ops = &virtio_gpu_vram_vm_ops; diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c b/drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c index 265f7c48d856..8c8015528b6f 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_ttm_glue.c @@ -97,7 +97,7 @@ int vmw_mmap(struct file *filp, struct vm_area_struct *vma) /* Use VM_PFNMAP rather than VM_MIXEDMAP if not a COW mapping */ if (!is_cow_mapping(vma->vm_flags)) - vma->vm_flags = (vma->vm_flags & ~VM_MIXEDMAP) | VM_PFNMAP; + mod_vm_flags(vma, VM_PFNMAP, VM_MIXEDMAP); ttm_bo_put(bo); /* release extra ref taken by ttm_bo_mmap_obj() */ diff --git a/drivers/gpu/drm/xen/xen_drm_front_gem.c b/drivers/gpu/drm/xen/xen_drm_front_gem.c index 4c95ebcdcc2d..18a93ad4aa1f 100644 --- a/drivers/gpu/drm/xen/xen_drm_front_gem.c +++ b/drivers/gpu/drm/xen/xen_drm_front_gem.c @@ -69,8 +69,7 @@ static int xen_drm_front_gem_object_mmap(struct drm_gem_object *gem_obj, * vm_pgoff (used as a fake buffer offset by DRM) to 0 as we want to map * the whole buffer. */ - vma->vm_flags &= ~VM_PFNMAP; - vma->vm_flags |= VM_MIXEDMAP | VM_DONTEXPAND; + mod_vm_flags(vma, VM_MIXEDMAP | VM_DONTEXPAND, VM_PFNMAP); vma->vm_pgoff = 0; /* diff --git a/drivers/hsi/clients/cmt_speech.c b/drivers/hsi/clients/cmt_speech.c index 8069f795c864..952a31e742a1 100644 --- a/drivers/hsi/clients/cmt_speech.c +++ b/drivers/hsi/clients/cmt_speech.c @@ -1264,7 +1264,7 @@ static int cs_char_mmap(struct file *file, struct vm_area_struct *vma) if (vma_pages(vma) != 1) return -EINVAL; - vma->vm_flags |= VM_IO | VM_DONTDUMP | VM_DONTEXPAND; + set_vm_flags(vma, VM_IO | VM_DONTDUMP | VM_DONTEXPAND); vma->vm_ops = &cs_char_vm_ops; vma->vm_private_data = file->private_data; diff --git a/drivers/hwtracing/intel_th/msu.c b/drivers/hwtracing/intel_th/msu.c index 6c8215a47a60..a6f178bf3ded 100644 --- a/drivers/hwtracing/intel_th/msu.c +++ b/drivers/hwtracing/intel_th/msu.c @@ -1659,7 +1659,7 @@ static int intel_th_msc_mmap(struct file *file, struct vm_area_struct *vma) atomic_dec(&msc->user_count); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_flags |= VM_DONTEXPAND | VM_DONTCOPY; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTCOPY); vma->vm_ops = &msc_mmap_ops; return ret; } diff --git a/drivers/hwtracing/stm/core.c b/drivers/hwtracing/stm/core.c index 2712e699ba08..9a59e61c4194 100644 --- a/drivers/hwtracing/stm/core.c +++ b/drivers/hwtracing/stm/core.c @@ -715,7 +715,7 @@ static int stm_char_mmap(struct file *file, struct vm_area_struct *vma) pm_runtime_get_sync(&stm->dev); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &stm_mmap_vmops; vm_iomap_memory(vma, phys, size); diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c index f5f9269fdc16..7294f2d33bc6 100644 --- a/drivers/infiniband/hw/hfi1/file_ops.c +++ b/drivers/infiniband/hw/hfi1/file_ops.c @@ -403,7 +403,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) ret = -EPERM; goto done; } - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); addr = vma->vm_start; for (i = 0 ; i < uctxt->egrbufs.numbufs; i++) { memlen = uctxt->egrbufs.buffers[i].len; @@ -528,7 +528,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma) goto done; } - vma->vm_flags = flags; + reset_vm_flags(vma, flags); hfi1_cdbg(PROC, "%u:%u type:%u io/vf:%d/%d, addr:0x%llx, len:%lu(%lu), flags:0x%lx\n", ctxt, subctxt, type, mapio, vmf, memaddr, memlen, diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index c669ef6e47e7..538318c809b3 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -2087,7 +2087,7 @@ static int mlx5_ib_mmap_clock_info_page(struct mlx5_ib_dev *dev, if (vma->vm_flags & (VM_WRITE | VM_EXEC)) return -EPERM; - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); if (!dev->mdev->clock_info) return -EOPNOTSUPP; @@ -2311,7 +2311,7 @@ static int mlx5_ib_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vm if (vma->vm_flags & VM_WRITE) return -EPERM; - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); /* Don't expose to user-space information it shouldn't have */ if (PAGE_SIZE > 4096) diff --git a/drivers/infiniband/hw/qib/qib_file_ops.c b/drivers/infiniband/hw/qib/qib_file_ops.c index 3937144b2ae5..16ef80df4b7f 100644 --- a/drivers/infiniband/hw/qib/qib_file_ops.c +++ b/drivers/infiniband/hw/qib/qib_file_ops.c @@ -733,7 +733,7 @@ static int qib_mmap_mem(struct vm_area_struct *vma, struct qib_ctxtdata *rcd, } /* don't allow them to later change with mprotect */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } pfn = virt_to_phys(kvaddr) >> PAGE_SHIFT; @@ -769,7 +769,7 @@ static int mmap_ureg(struct vm_area_struct *vma, struct qib_devdata *dd, phys = dd->physaddr + ureg; vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND); ret = io_remap_pfn_range(vma, vma->vm_start, phys >> PAGE_SHIFT, vma->vm_end - vma->vm_start, @@ -810,8 +810,7 @@ static int mmap_piobufs(struct vm_area_struct *vma, * don't allow them to later change to readable with mprotect (for when * not initially mapped readable, as is normally the case) */ - vma->vm_flags &= ~VM_MAYREAD; - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND; + mod_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND, VM_MAYREAD); /* We used PAT if wc_cookie == 0 */ if (!dd->wc_cookie) @@ -852,7 +851,7 @@ static int mmap_rcvegrbufs(struct vm_area_struct *vma, goto bail; } /* don't allow them to later change to writable with mprotect */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); start = vma->vm_start; @@ -944,7 +943,7 @@ static int mmap_kvaddr(struct vm_area_struct *vma, u64 pgaddr, * Don't allow permission to later change to writable * with mprotect. */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } else goto bail; len = vma->vm_end - vma->vm_start; @@ -955,7 +954,7 @@ static int mmap_kvaddr(struct vm_area_struct *vma, u64 pgaddr, vma->vm_pgoff = (unsigned long) addr >> PAGE_SHIFT; vma->vm_ops = &qib_file_vm_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); ret = 1; bail: diff --git a/drivers/infiniband/hw/usnic/usnic_ib_verbs.c b/drivers/infiniband/hw/usnic/usnic_ib_verbs.c index 6e8c4fbb8083..6f9237c2a26b 100644 --- a/drivers/infiniband/hw/usnic/usnic_ib_verbs.c +++ b/drivers/infiniband/hw/usnic/usnic_ib_verbs.c @@ -672,7 +672,7 @@ int usnic_ib_mmap(struct ib_ucontext *context, usnic_dbg("\n"); us_ibdev = to_usdev(context->device); - vma->vm_flags |= VM_IO; + set_vm_flags(vma, VM_IO); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vfid = vma->vm_pgoff; usnic_dbg("Page Offset %lu PAGE_SHIFT %u VFID %u\n", diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c index 19176583dbde..7f1b7b5dd3f4 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_verbs.c @@ -408,7 +408,7 @@ int pvrdma_mmap(struct ib_ucontext *ibcontext, struct vm_area_struct *vma) } /* Map UAR to kernel space, VM_LOCKED? */ - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); if (io_remap_pfn_range(vma, start, context->uar.pfn, size, vma->vm_page_prot)) diff --git a/drivers/media/common/videobuf2/videobuf2-dma-contig.c b/drivers/media/common/videobuf2/videobuf2-dma-contig.c index 5f1175f8b349..e66ae399749e 100644 --- a/drivers/media/common/videobuf2/videobuf2-dma-contig.c +++ b/drivers/media/common/videobuf2/videobuf2-dma-contig.c @@ -293,7 +293,7 @@ static int vb2_dc_mmap(void *buf_priv, struct vm_area_struct *vma) return ret; } - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_private_data = &buf->handler; vma->vm_ops = &vb2_common_vm_ops; diff --git a/drivers/media/common/videobuf2/videobuf2-vmalloc.c b/drivers/media/common/videobuf2/videobuf2-vmalloc.c index 959b45beb1f3..edb47240ec17 100644 --- a/drivers/media/common/videobuf2/videobuf2-vmalloc.c +++ b/drivers/media/common/videobuf2/videobuf2-vmalloc.c @@ -185,7 +185,7 @@ static int vb2_vmalloc_mmap(void *buf_priv, struct vm_area_struct *vma) /* * Make sure that vm_areas for 2 buffers won't be merged together */ - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); /* * Use common vm_area operations to track buffer refcount. diff --git a/drivers/media/v4l2-core/videobuf-dma-contig.c b/drivers/media/v4l2-core/videobuf-dma-contig.c index f2c439359557..c030823185ba 100644 --- a/drivers/media/v4l2-core/videobuf-dma-contig.c +++ b/drivers/media/v4l2-core/videobuf-dma-contig.c @@ -314,7 +314,7 @@ static int __videobuf_mmap_mapper(struct videobuf_queue *q, } vma->vm_ops = &videobuf_vm_ops; - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); vma->vm_private_data = map; dev_dbg(q->dev, "mmap %p: q=%p %08lx-%08lx (%lx) pgoff %08lx buf %d\n", diff --git a/drivers/media/v4l2-core/videobuf-dma-sg.c b/drivers/media/v4l2-core/videobuf-dma-sg.c index 234e9f647c96..9adac4875f29 100644 --- a/drivers/media/v4l2-core/videobuf-dma-sg.c +++ b/drivers/media/v4l2-core/videobuf-dma-sg.c @@ -630,8 +630,8 @@ static int __videobuf_mmap_mapper(struct videobuf_queue *q, map->count = 1; map->q = q; vma->vm_ops = &videobuf_vm_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; - vma->vm_flags &= ~VM_IO; /* using shared anonymous pages */ + /* using shared anonymous pages */ + mod_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP, VM_IO); vma->vm_private_data = map; dprintk(1, "mmap %p: q=%p %08lx-%08lx pgoff %08lx bufs %d-%d\n", map, q, vma->vm_start, vma->vm_end, vma->vm_pgoff, first, last); diff --git a/drivers/media/v4l2-core/videobuf-vmalloc.c b/drivers/media/v4l2-core/videobuf-vmalloc.c index 9b2443720ab0..48d439ccd414 100644 --- a/drivers/media/v4l2-core/videobuf-vmalloc.c +++ b/drivers/media/v4l2-core/videobuf-vmalloc.c @@ -247,7 +247,7 @@ static int __videobuf_mmap_mapper(struct videobuf_queue *q, } vma->vm_ops = &videobuf_vm_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_private_data = map; dprintk(1, "mmap %p: q=%p %08lx-%08lx (%lx) pgoff %08lx buf %d\n", diff --git a/drivers/misc/cxl/context.c b/drivers/misc/cxl/context.c index acaa44809c58..17562e4efcb2 100644 --- a/drivers/misc/cxl/context.c +++ b/drivers/misc/cxl/context.c @@ -220,7 +220,7 @@ int cxl_context_iomap(struct cxl_context *ctx, struct vm_area_struct *vma) pr_devel("%s: mmio physical: %llx pe: %i master:%i\n", __func__, ctx->psn_phys, ctx->pe , ctx->master); - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &cxl_mmap_vmops; return 0; diff --git a/drivers/misc/habanalabs/common/memory.c b/drivers/misc/habanalabs/common/memory.c index 5e9ae7600d75..ad8eae764b9b 100644 --- a/drivers/misc/habanalabs/common/memory.c +++ b/drivers/misc/habanalabs/common/memory.c @@ -2082,7 +2082,7 @@ static int hl_ts_mmap(struct hl_mmap_mem_buf *buf, struct vm_area_struct *vma, v { struct hl_ts_buff *ts_buff = buf->private; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP | VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP | VM_DONTCOPY | VM_NORESERVE); return remap_vmalloc_range(vma, ts_buff->user_buff_address, 0); } diff --git a/drivers/misc/habanalabs/gaudi/gaudi.c b/drivers/misc/habanalabs/gaudi/gaudi.c index 9f5e208701ba..4186f04da224 100644 --- a/drivers/misc/habanalabs/gaudi/gaudi.c +++ b/drivers/misc/habanalabs/gaudi/gaudi.c @@ -4236,8 +4236,8 @@ static int gaudi_mmap(struct hl_device *hdev, struct vm_area_struct *vma, { int rc; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | - VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | + VM_DONTCOPY | VM_NORESERVE); rc = dma_mmap_coherent(hdev->dev, vma, cpu_addr, (dma_addr - HOST_PHYS_BASE), size); diff --git a/drivers/misc/habanalabs/gaudi2/gaudi2.c b/drivers/misc/habanalabs/gaudi2/gaudi2.c index e793fb2bdcbe..7311c3053944 100644 --- a/drivers/misc/habanalabs/gaudi2/gaudi2.c +++ b/drivers/misc/habanalabs/gaudi2/gaudi2.c @@ -5538,8 +5538,8 @@ static int gaudi2_mmap(struct hl_device *hdev, struct vm_area_struct *vma, { int rc; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | - VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | + VM_DONTCOPY | VM_NORESERVE); #ifdef _HAS_DMA_MMAP_COHERENT @@ -10116,8 +10116,8 @@ static int gaudi2_block_mmap(struct hl_device *hdev, struct vm_area_struct *vma, address = pci_resource_start(hdev->pdev, SRAM_CFG_BAR_ID) + offset_in_bar; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | - VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | + VM_DONTCOPY | VM_NORESERVE); rc = remap_pfn_range(vma, vma->vm_start, address >> PAGE_SHIFT, block_size, vma->vm_page_prot); diff --git a/drivers/misc/habanalabs/goya/goya.c b/drivers/misc/habanalabs/goya/goya.c index 0f083fcf81a6..5e2aaa26ea29 100644 --- a/drivers/misc/habanalabs/goya/goya.c +++ b/drivers/misc/habanalabs/goya/goya.c @@ -2880,8 +2880,8 @@ static int goya_mmap(struct hl_device *hdev, struct vm_area_struct *vma, { int rc; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | - VM_DONTCOPY | VM_NORESERVE; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP | + VM_DONTCOPY | VM_NORESERVE); rc = dma_mmap_coherent(hdev->dev, vma, cpu_addr, (dma_addr - HOST_PHYS_BASE), size); diff --git a/drivers/misc/ocxl/context.c b/drivers/misc/ocxl/context.c index 9eb0d93b01c6..e6f941248e93 100644 --- a/drivers/misc/ocxl/context.c +++ b/drivers/misc/ocxl/context.c @@ -180,7 +180,7 @@ static int check_mmap_afu_irq(struct ocxl_context *ctx, if ((vma->vm_flags & VM_READ) || (vma->vm_flags & VM_EXEC) || !(vma->vm_flags & VM_WRITE)) return -EINVAL; - vma->vm_flags &= ~(VM_MAYREAD | VM_MAYEXEC); + clear_vm_flags(vma, VM_MAYREAD | VM_MAYEXEC); return 0; } @@ -204,7 +204,7 @@ int ocxl_context_mmap(struct ocxl_context *ctx, struct vm_area_struct *vma) if (rc) return rc; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &ocxl_vmops; return 0; diff --git a/drivers/misc/ocxl/sysfs.c b/drivers/misc/ocxl/sysfs.c index 25c78df8055d..9398246cac79 100644 --- a/drivers/misc/ocxl/sysfs.c +++ b/drivers/misc/ocxl/sysfs.c @@ -134,7 +134,7 @@ static int global_mmio_mmap(struct file *filp, struct kobject *kobj, (afu->config.global_mmio_size >> PAGE_SHIFT)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &global_mmio_vmops; vma->vm_private_data = afu; diff --git a/drivers/misc/open-dice.c b/drivers/misc/open-dice.c index c61be3404c6f..9f9438b5b075 100644 --- a/drivers/misc/open-dice.c +++ b/drivers/misc/open-dice.c @@ -96,13 +96,13 @@ static int open_dice_mmap(struct file *filp, struct vm_area_struct *vma) /* Ensure userspace cannot acquire VM_WRITE + VM_SHARED later. */ if (vma->vm_flags & VM_WRITE) - vma->vm_flags &= ~VM_MAYSHARE; + clear_vm_flags(vma, VM_MAYSHARE); else if (vma->vm_flags & VM_SHARED) - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); /* Create write-combine mapping so all clients observe a wipe. */ vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); - vma->vm_flags |= VM_DONTCOPY | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTDUMP); return vm_iomap_memory(vma, drvdata->rmem->base, drvdata->rmem->size); } diff --git a/drivers/misc/sgi-gru/grufile.c b/drivers/misc/sgi-gru/grufile.c index 7ffcfc0bb587..8b777286d3b2 100644 --- a/drivers/misc/sgi-gru/grufile.c +++ b/drivers/misc/sgi-gru/grufile.c @@ -101,8 +101,8 @@ static int gru_file_mmap(struct file *file, struct vm_area_struct *vma) vma->vm_end & (GRU_GSEG_PAGESIZE - 1)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_LOCKED | - VM_DONTCOPY | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_LOCKED | + VM_DONTCOPY | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = PAGE_SHARED; vma->vm_ops = &gru_vm_ops; diff --git a/drivers/misc/uacce/uacce.c b/drivers/misc/uacce/uacce.c index 905eff1f840e..f57e91cdb0f6 100644 --- a/drivers/misc/uacce/uacce.c +++ b/drivers/misc/uacce/uacce.c @@ -229,7 +229,7 @@ static int uacce_fops_mmap(struct file *filep, struct vm_area_struct *vma) if (!qfr) return -ENOMEM; - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND | VM_WIPEONFORK; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND | VM_WIPEONFORK); vma->vm_ops = &uacce_vm_ops; vma->vm_private_data = q; qfr->type = type; diff --git a/drivers/sbus/char/oradax.c b/drivers/sbus/char/oradax.c index 21b7cb6e7e70..a096734daad0 100644 --- a/drivers/sbus/char/oradax.c +++ b/drivers/sbus/char/oradax.c @@ -389,7 +389,7 @@ static int dax_devmap(struct file *f, struct vm_area_struct *vma) /* completion area is mapped read-only for user */ if (vma->vm_flags & VM_WRITE) return -EPERM; - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); if (remap_pfn_range(vma, vma->vm_start, ctx->ca_buf_ra >> PAGE_SHIFT, len, vma->vm_page_prot)) diff --git a/drivers/scsi/cxlflash/ocxl_hw.c b/drivers/scsi/cxlflash/ocxl_hw.c index 631eda2d467e..d386c25c2699 100644 --- a/drivers/scsi/cxlflash/ocxl_hw.c +++ b/drivers/scsi/cxlflash/ocxl_hw.c @@ -1167,7 +1167,7 @@ static int afu_mmap(struct file *file, struct vm_area_struct *vma) (ctx->psn_size >> PAGE_SHIFT)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP; + set_vm_flags(vma, VM_IO | VM_PFNMAP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); vma->vm_ops = &ocxlflash_vmops; return 0; diff --git a/drivers/scsi/sg.c b/drivers/scsi/sg.c index ff9854f59964..7438adfe3bdc 100644 --- a/drivers/scsi/sg.c +++ b/drivers/scsi/sg.c @@ -1288,7 +1288,7 @@ sg_mmap(struct file *filp, struct vm_area_struct *vma) } sfp->mmap_called = 1; - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_private_data = sfp; vma->vm_ops = &sg_mmap_vm_ops; out: diff --git a/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c b/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c index 5e53eed8ae95..df1c944e5058 100644 --- a/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c +++ b/drivers/staging/media/atomisp/pci/hmm/hmm_bo.c @@ -1072,7 +1072,7 @@ int hmm_bo_mmap(struct vm_area_struct *vma, struct hmm_buffer_object *bo) vma->vm_private_data = bo; vma->vm_ops = &hmm_bo_vm_ops; - vma->vm_flags |= VM_IO | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); /* * call hmm_bo_vm_open explicitly. diff --git a/drivers/staging/media/deprecated/meye/meye.c b/drivers/staging/media/deprecated/meye/meye.c index 5d87efd9b95c..2505e64d7119 100644 --- a/drivers/staging/media/deprecated/meye/meye.c +++ b/drivers/staging/media/deprecated/meye/meye.c @@ -1476,8 +1476,8 @@ static int meye_mmap(struct file *file, struct vm_area_struct *vma) } vma->vm_ops = &meye_vm_ops; - vma->vm_flags &= ~VM_IO; /* not I/O memory */ - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + /* not I/O memory */ + mod_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP, VM_IO); vma->vm_private_data = (void *) (offset / gbufsize); meye_vm_open(vma); diff --git a/drivers/staging/media/deprecated/stkwebcam/stk-webcam.c b/drivers/staging/media/deprecated/stkwebcam/stk-webcam.c index 787edb3d47c2..196d1034f104 100644 --- a/drivers/staging/media/deprecated/stkwebcam/stk-webcam.c +++ b/drivers/staging/media/deprecated/stkwebcam/stk-webcam.c @@ -779,7 +779,7 @@ static int v4l_stk_mmap(struct file *fp, struct vm_area_struct *vma) ret = remap_vmalloc_range(vma, sbuf->buffer, 0); if (ret) return ret; - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); vma->vm_private_data = sbuf; vma->vm_ops = &stk_v4l_vm_ops; sbuf->v4lbuf.flags |= V4L2_BUF_FLAG_MAPPED; diff --git a/drivers/target/target_core_user.c b/drivers/target/target_core_user.c index 2940559c3086..9fd64259904c 100644 --- a/drivers/target/target_core_user.c +++ b/drivers/target/target_core_user.c @@ -1928,7 +1928,7 @@ static int tcmu_mmap(struct uio_info *info, struct vm_area_struct *vma) { struct tcmu_dev *udev = container_of(info, struct tcmu_dev, uio_info); - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &tcmu_vm_ops; vma->vm_private_data = udev; diff --git a/drivers/uio/uio.c b/drivers/uio/uio.c index 43afbb7c5ab9..08802744f3b7 100644 --- a/drivers/uio/uio.c +++ b/drivers/uio/uio.c @@ -713,7 +713,7 @@ static const struct vm_operations_struct uio_logical_vm_ops = { static int uio_mmap_logical(struct vm_area_struct *vma) { - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &uio_logical_vm_ops; return 0; } diff --git a/drivers/usb/core/devio.c b/drivers/usb/core/devio.c index 837f3e57f580..d9aefa259883 100644 --- a/drivers/usb/core/devio.c +++ b/drivers/usb/core/devio.c @@ -279,8 +279,7 @@ static int usbdev_mmap(struct file *file, struct vm_area_struct *vma) } } - vma->vm_flags |= VM_IO; - vma->vm_flags |= (VM_DONTEXPAND | VM_DONTDUMP); + set_vm_flags(vma, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &usbdev_vm_ops; vma->vm_private_data = usbm; diff --git a/drivers/usb/mon/mon_bin.c b/drivers/usb/mon/mon_bin.c index 094e812e9e69..9b2d48a65fdf 100644 --- a/drivers/usb/mon/mon_bin.c +++ b/drivers/usb/mon/mon_bin.c @@ -1272,8 +1272,7 @@ static int mon_bin_mmap(struct file *filp, struct vm_area_struct *vma) if (vma->vm_flags & VM_WRITE) return -EPERM; - vma->vm_flags &= ~VM_MAYWRITE; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + mod_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP, VM_MAYWRITE); vma->vm_private_data = filp->private_data; mon_bin_vma_open(vma); return 0; diff --git a/drivers/vdpa/vdpa_user/iova_domain.c b/drivers/vdpa/vdpa_user/iova_domain.c index e682bc7ee6c9..39dcce2e455b 100644 --- a/drivers/vdpa/vdpa_user/iova_domain.c +++ b/drivers/vdpa/vdpa_user/iova_domain.c @@ -512,7 +512,7 @@ static int vduse_domain_mmap(struct file *file, struct vm_area_struct *vma) { struct vduse_iova_domain *domain = file->private_data; - vma->vm_flags |= VM_DONTDUMP | VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTDUMP | VM_DONTEXPAND); vma->vm_private_data = domain; vma->vm_ops = &vduse_domain_mmap_ops; diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index 26a541cc64d1..86eb3fc9ffb4 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1799,7 +1799,7 @@ int vfio_pci_core_mmap(struct vfio_device *core_vdev, struct vm_area_struct *vma * See remap_pfn_range(), called from vfio_pci_fault() but we can't * change vm_flags within the fault handler. Set them now. */ - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &vfio_pci_mmap_ops; return 0; diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index ec32f785dfde..7b81994a7d02 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -1315,7 +1315,7 @@ static int vhost_vdpa_mmap(struct file *file, struct vm_area_struct *vma) if (vma->vm_end - vma->vm_start != notify.size) return -ENOTSUPP; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &vhost_vdpa_vm_ops; return 0; } diff --git a/drivers/video/fbdev/68328fb.c b/drivers/video/fbdev/68328fb.c index 7db03ed77c76..a794a740af10 100644 --- a/drivers/video/fbdev/68328fb.c +++ b/drivers/video/fbdev/68328fb.c @@ -391,7 +391,7 @@ static int mc68x328fb_mmap(struct fb_info *info, struct vm_area_struct *vma) #ifndef MMU /* this is uClinux (no MMU) specific code */ - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_start = videomemory; return 0; diff --git a/drivers/video/fbdev/core/fb_defio.c b/drivers/video/fbdev/core/fb_defio.c index c730253ab85c..af0bfaa2d014 100644 --- a/drivers/video/fbdev/core/fb_defio.c +++ b/drivers/video/fbdev/core/fb_defio.c @@ -232,9 +232,9 @@ static const struct address_space_operations fb_deferred_io_aops = { int fb_deferred_io_mmap(struct fb_info *info, struct vm_area_struct *vma) { vma->vm_ops = &fb_deferred_io_vm_ops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); if (!(info->flags & FBINFO_VIRTFB)) - vma->vm_flags |= VM_IO; + set_vm_flags(vma, VM_IO); vma->vm_private_data = info; return 0; } diff --git a/drivers/xen/gntalloc.c b/drivers/xen/gntalloc.c index a15729beb9d1..ee4a8958dc68 100644 --- a/drivers/xen/gntalloc.c +++ b/drivers/xen/gntalloc.c @@ -525,7 +525,7 @@ static int gntalloc_mmap(struct file *filp, struct vm_area_struct *vma) vma->vm_private_data = vm_priv; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &gntalloc_vmops; diff --git a/drivers/xen/gntdev.c b/drivers/xen/gntdev.c index 4d9a3050de6a..6d5bb1ebb661 100644 --- a/drivers/xen/gntdev.c +++ b/drivers/xen/gntdev.c @@ -1055,10 +1055,10 @@ static int gntdev_mmap(struct file *flip, struct vm_area_struct *vma) vma->vm_ops = &gntdev_vmops; - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP | VM_MIXEDMAP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP | VM_MIXEDMAP); if (use_ptemod) - vma->vm_flags |= VM_DONTCOPY; + set_vm_flags(vma, VM_DONTCOPY); vma->vm_private_data = map; if (map->flags) { diff --git a/drivers/xen/privcmd-buf.c b/drivers/xen/privcmd-buf.c index dd5bbb6e1b6b..037547918630 100644 --- a/drivers/xen/privcmd-buf.c +++ b/drivers/xen/privcmd-buf.c @@ -156,7 +156,7 @@ static int privcmd_buf_mmap(struct file *file, struct vm_area_struct *vma) vma_priv->file_priv = file_priv; vma_priv->users = 1; - vma->vm_flags |= VM_IO | VM_DONTEXPAND; + set_vm_flags(vma, VM_IO | VM_DONTEXPAND); vma->vm_ops = &privcmd_buf_vm_ops; vma->vm_private_data = vma_priv; diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c index 1edf45ee9890..4c8cfc6f86d8 100644 --- a/drivers/xen/privcmd.c +++ b/drivers/xen/privcmd.c @@ -934,8 +934,8 @@ static int privcmd_mmap(struct file *file, struct vm_area_struct *vma) { /* DONTCOPY is essential for Xen because copy_page_range doesn't know * how to recreate these mappings */ - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTCOPY | - VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTCOPY | + VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &privcmd_vm_ops; vma->vm_private_data = NULL; diff --git a/fs/aio.c b/fs/aio.c index 562916d85cba..db821fb1e92d 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -390,7 +390,7 @@ static const struct vm_operations_struct aio_ring_vm_ops = { static int aio_ring_mmap(struct file *file, struct vm_area_struct *vma) { - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); vma->vm_ops = &aio_ring_vm_ops; return 0; } diff --git a/fs/cramfs/inode.c b/fs/cramfs/inode.c index 61ccf7722fc3..874a17a1b8d9 100644 --- a/fs/cramfs/inode.c +++ b/fs/cramfs/inode.c @@ -408,7 +408,7 @@ static int cramfs_physmem_mmap(struct file *file, struct vm_area_struct *vma) * unpopulated ptes via cramfs_read_folio(). */ int i; - vma->vm_flags |= VM_MIXEDMAP; + set_vm_flags(vma, VM_MIXEDMAP); for (i = 0; i < pages && !ret; i++) { vm_fault_t vmf; unsigned long off = i * PAGE_SIZE; diff --git a/fs/erofs/data.c b/fs/erofs/data.c index f57f921683d7..e6413ced2bb1 100644 --- a/fs/erofs/data.c +++ b/fs/erofs/data.c @@ -429,7 +429,7 @@ static int erofs_file_mmap(struct file *file, struct vm_area_struct *vma) return -EINVAL; vma->vm_ops = &erofs_dax_vm_ops; - vma->vm_flags |= VM_HUGEPAGE; + set_vm_flags(vma, VM_HUGEPAGE); return 0; } #else diff --git a/fs/exec.c b/fs/exec.c index ab913243a367..5e1631e109a8 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -270,7 +270,7 @@ static int __bprm_mm_init(struct linux_binprm *bprm) BUILD_BUG_ON(VM_STACK_FLAGS & VM_STACK_INCOMPLETE_SETUP); vma->vm_end = STACK_TOP_MAX; vma->vm_start = vma->vm_end - PAGE_SIZE; - vma->vm_flags = VM_SOFTDIRTY | VM_STACK_FLAGS | VM_STACK_INCOMPLETE_SETUP; + init_vm_flags(vma, VM_SOFTDIRTY | VM_STACK_FLAGS | VM_STACK_INCOMPLETE_SETUP); vma->vm_page_prot = vm_get_page_prot(vma->vm_flags); err = insert_vm_struct(mm, vma); @@ -834,7 +834,7 @@ int setup_arg_pages(struct linux_binprm *bprm, } /* mprotect_fixup is overkill to remove the temporary stack flags */ - vma->vm_flags &= ~VM_STACK_INCOMPLETE_SETUP; + clear_vm_flags(vma, VM_STACK_INCOMPLETE_SETUP); stack_expand = 131072UL; /* randomly 32*4k (or 2*64k) pages */ stack_size = vma->vm_end - vma->vm_start; diff --git a/fs/ext4/file.c b/fs/ext4/file.c index 7ac0a81bd371..baeb385b07c7 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -801,7 +801,7 @@ static int ext4_file_mmap(struct file *file, struct vm_area_struct *vma) file_accessed(file); if (IS_DAX(file_inode(file))) { vma->vm_ops = &ext4_dax_vm_ops; - vma->vm_flags |= VM_HUGEPAGE; + set_vm_flags(vma, VM_HUGEPAGE); } else { vma->vm_ops = &ext4_file_vm_ops; } diff --git a/fs/fuse/dax.c b/fs/fuse/dax.c index e23e802a8013..599969edc869 100644 --- a/fs/fuse/dax.c +++ b/fs/fuse/dax.c @@ -860,7 +860,7 @@ int fuse_dax_mmap(struct file *file, struct vm_area_struct *vma) { file_accessed(file); vma->vm_ops = &fuse_dax_vm_ops; - vma->vm_flags |= VM_MIXEDMAP | VM_HUGEPAGE; + set_vm_flags(vma, VM_MIXEDMAP | VM_HUGEPAGE); return 0; } diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 790d2727141a..d63a392985a7 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -132,7 +132,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma) * way when do_mmap unwinds (may be important on powerpc * and ia64). */ - vma->vm_flags |= VM_HUGETLB | VM_DONTEXPAND; + set_vm_flags(vma, VM_HUGETLB | VM_DONTEXPAND); vma->vm_ops = &hugetlb_vm_ops; ret = seal_check_future_write(info->seals, vma); @@ -813,7 +813,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset, * as input to create an allocation policy. */ vma_init(&pseudo_vma, mm); - pseudo_vma.vm_flags = (VM_HUGETLB | VM_MAYSHARE | VM_SHARED); + init_vm_flags(&pseudo_vma, VM_HUGETLB | VM_MAYSHARE | VM_SHARED); pseudo_vma.vm_file = file; for (index = start; index < end; index++) { diff --git a/fs/orangefs/file.c b/fs/orangefs/file.c index 167fa43b24f9..0f668db6bcf3 100644 --- a/fs/orangefs/file.c +++ b/fs/orangefs/file.c @@ -389,8 +389,7 @@ static int orangefs_file_mmap(struct file *file, struct vm_area_struct *vma) "orangefs_file_mmap: called on %pD\n", file); /* set the sequential readahead hint */ - vma->vm_flags |= VM_SEQ_READ; - vma->vm_flags &= ~VM_RAND_READ; + mod_vm_flags(vma, VM_SEQ_READ, VM_RAND_READ); file_accessed(file); vma->vm_ops = &orangefs_file_vm_ops; diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index e35a0398db63..4d651777c8a5 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1302,7 +1302,7 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf, mas_for_each(&mas, vma, ULONG_MAX) { if (!(vma->vm_flags & VM_SOFTDIRTY)) continue; - vma->vm_flags &= ~VM_SOFTDIRTY; + clear_vm_flags(vma, VM_SOFTDIRTY); vma_set_page_prot(vma); } diff --git a/fs/proc/vmcore.c b/fs/proc/vmcore.c index 09a81e4b1273..858e4e804f85 100644 --- a/fs/proc/vmcore.c +++ b/fs/proc/vmcore.c @@ -582,8 +582,7 @@ static int mmap_vmcore(struct file *file, struct vm_area_struct *vma) if (vma->vm_flags & (VM_WRITE | VM_EXEC)) return -EPERM; - vma->vm_flags &= ~(VM_MAYWRITE | VM_MAYEXEC); - vma->vm_flags |= VM_MIXEDMAP; + mod_vm_flags(vma, VM_MIXEDMAP, VM_MAYWRITE | VM_MAYEXEC); vma->vm_ops = &vmcore_mmap_ops; len = 0; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 98ac37e34e3d..f46252544924 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -618,7 +618,7 @@ static void userfaultfd_event_wait_completion(struct userfaultfd_ctx *ctx, for_each_vma(vmi, vma) { if (vma->vm_userfaultfd_ctx.ctx == release_new_ctx) { vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; - vma->vm_flags &= ~__VM_UFFD_FLAGS; + clear_vm_flags(vma, __VM_UFFD_FLAGS); } } mmap_write_unlock(mm); @@ -652,7 +652,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs) octx = vma->vm_userfaultfd_ctx.ctx; if (!octx || !(octx->features & UFFD_FEATURE_EVENT_FORK)) { vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; - vma->vm_flags &= ~__VM_UFFD_FLAGS; + clear_vm_flags(vma, __VM_UFFD_FLAGS); return 0; } @@ -733,7 +733,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma, } else { /* Drop uffd context if remap feature not enabled */ vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; - vma->vm_flags &= ~__VM_UFFD_FLAGS; + clear_vm_flags(vma, __VM_UFFD_FLAGS); } } @@ -895,7 +895,7 @@ static int userfaultfd_release(struct inode *inode, struct file *file) prev = vma; } - vma->vm_flags = new_flags; + reset_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; } mmap_write_unlock(mm); @@ -1463,7 +1463,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ - vma->vm_flags = new_flags; + reset_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx.ctx = ctx; if (is_vm_hugetlb_page(vma) && uffd_disable_huge_pmd_share(vma)) @@ -1651,7 +1651,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * the next vma was merged into the current one and * the current one has not been updated yet. */ - vma->vm_flags = new_flags; + reset_vm_flags(vma, new_flags); vma->vm_userfaultfd_ctx = NULL_VM_UFFD_CTX; skip: diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index 595a5bcf46b9..bf777fed0dd4 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1429,7 +1429,7 @@ xfs_file_mmap( file_accessed(file); vma->vm_ops = &xfs_file_vm_ops; if (IS_DAX(inode)) - vma->vm_flags |= VM_HUGEPAGE; + set_vm_flags(vma, VM_HUGEPAGE); return 0; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 2b16d45b75a6..594e835bad9c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3756,7 +3756,7 @@ static inline int seal_check_future_write(int seals, struct vm_area_struct *vma) * VM_MAYWRITE as we still want them to be COW-writable. */ if (vma->vm_flags & VM_SHARED) - vma->vm_flags &= ~(VM_MAYWRITE); + clear_vm_flags(vma, VM_MAYWRITE); } return 0; diff --git a/kernel/bpf/ringbuf.c b/kernel/bpf/ringbuf.c index 80f4b4d88aaf..d2c967cc2873 100644 --- a/kernel/bpf/ringbuf.c +++ b/kernel/bpf/ringbuf.c @@ -269,7 +269,7 @@ static int ringbuf_map_mmap_kern(struct bpf_map *map, struct vm_area_struct *vma if (vma->vm_pgoff != 0 || vma->vm_end - vma->vm_start != PAGE_SIZE) return -EPERM; } else { - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } /* remap_vmalloc_range() checks size and offset constraints */ return remap_vmalloc_range(vma, rb_map->rb, @@ -290,7 +290,7 @@ static int ringbuf_map_mmap_user(struct bpf_map *map, struct vm_area_struct *vma */ return -EPERM; } else { - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); } /* remap_vmalloc_range() checks size and offset constraints */ return remap_vmalloc_range(vma, rb_map->rb, vma->vm_pgoff + RINGBUF_PGOFF); diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 64131f88c553..db19094c7ac7 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -882,10 +882,10 @@ static int bpf_map_mmap(struct file *filp, struct vm_area_struct *vma) /* set default open/close callbacks */ vma->vm_ops = &bpf_map_default_vmops; vma->vm_private_data = map; - vma->vm_flags &= ~VM_MAYEXEC; + clear_vm_flags(vma, VM_MAYEXEC); if (!(vma->vm_flags & VM_WRITE)) /* disallow re-mapping with PROT_WRITE */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); err = map->ops->map_mmap(map, vma); if (err) diff --git a/kernel/events/core.c b/kernel/events/core.c index d56328e5080e..6745460dcf49 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -6573,7 +6573,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) * Since pinned accounting is per vm we cannot allow fork() to copy our * vma. */ - vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTCOPY | VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &perf_mmap_vmops; if (event->pmu->event_mapped) diff --git a/kernel/kcov.c b/kernel/kcov.c index e5cd09fd8a05..27fc1e26e1e1 100644 --- a/kernel/kcov.c +++ b/kernel/kcov.c @@ -489,7 +489,7 @@ static int kcov_mmap(struct file *filep, struct vm_area_struct *vma) goto exit; } spin_unlock_irqrestore(&kcov->lock, flags); - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); for (off = 0; off < size; off += PAGE_SIZE) { page = vmalloc_to_page(kcov->area + off); res = vm_insert_page(vma, vma->vm_start + off, page); diff --git a/kernel/relay.c b/kernel/relay.c index ef12532168d9..085aa8707bc2 100644 --- a/kernel/relay.c +++ b/kernel/relay.c @@ -91,7 +91,7 @@ static int relay_mmap_buf(struct rchan_buf *buf, struct vm_area_struct *vma) return -EINVAL; vma->vm_ops = &relay_file_mmap_ops; - vma->vm_flags |= VM_DONTEXPAND; + set_vm_flags(vma, VM_DONTEXPAND); vma->vm_private_data = buf; return 0; diff --git a/mm/madvise.c b/mm/madvise.c index a56a6d17e201..5b74321bcac9 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -179,7 +179,7 @@ static int madvise_update_vma(struct vm_area_struct *vma, /* * vm_flags is protected by the mmap_lock held in write mode. */ - vma->vm_flags = new_flags; + reset_vm_flags(vma, new_flags); if (!vma->vm_file || vma_is_anon_shmem(vma)) { error = replace_anon_vma_name(vma, anon_name); if (error) diff --git a/mm/memory.c b/mm/memory.c index aad226daf41b..2fabf89b2be9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1951,7 +1951,7 @@ int vm_insert_pages(struct vm_area_struct *vma, unsigned long addr, if (!(vma->vm_flags & VM_MIXEDMAP)) { BUG_ON(mmap_read_trylock(vma->vm_mm)); BUG_ON(vma->vm_flags & VM_PFNMAP); - vma->vm_flags |= VM_MIXEDMAP; + set_vm_flags(vma, VM_MIXEDMAP); } /* Defer page refcount checking till we're about to map that page. */ return insert_pages(vma, addr, pages, num, vma->vm_page_prot); @@ -2009,7 +2009,7 @@ int vm_insert_page(struct vm_area_struct *vma, unsigned long addr, if (!(vma->vm_flags & VM_MIXEDMAP)) { BUG_ON(mmap_read_trylock(vma->vm_mm)); BUG_ON(vma->vm_flags & VM_PFNMAP); - vma->vm_flags |= VM_MIXEDMAP; + set_vm_flags(vma, VM_MIXEDMAP); } return insert_page(vma, addr, page, vma->vm_page_prot); } @@ -2475,7 +2475,7 @@ int remap_pfn_range_notrack(struct vm_area_struct *vma, unsigned long addr, vma->vm_pgoff = pfn; } - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); BUG_ON(addr >= end); pfn -= addr >> PAGE_SHIFT; diff --git a/mm/mlock.c b/mm/mlock.c index 06aa9e204fac..4807e91aaa8b 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -380,7 +380,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, */ if (newflags & VM_LOCKED) newflags |= VM_IO; - WRITE_ONCE(vma->vm_flags, newflags); + reset_vm_flags(vma, newflags); lru_add_drain(); walk_page_range(vma->vm_mm, start, end, &mlock_walk_ops, NULL); @@ -388,7 +388,7 @@ static void mlock_vma_pages_range(struct vm_area_struct *vma, if (newflags & VM_IO) { newflags &= ~VM_IO; - WRITE_ONCE(vma->vm_flags, newflags); + reset_vm_flags(vma, newflags); } } @@ -456,7 +456,7 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, if ((newflags & VM_LOCKED) && (oldflags & VM_LOCKED)) { /* No work to do, and mlocking twice would be wrong */ - vma->vm_flags = newflags; + reset_vm_flags(vma, newflags); } else { mlock_vma_pages_range(vma, start, end, newflags); } diff --git a/mm/mmap.c b/mm/mmap.c index 5c4b608edde9..fa994ae903d9 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2607,7 +2607,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_start = addr; vma->vm_end = end; - vma->vm_flags = vm_flags; + init_vm_flags(vma, vm_flags); vma->vm_page_prot = vm_get_page_prot(vm_flags); vma->vm_pgoff = pgoff; @@ -2736,7 +2736,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, * then new mapped in-place (which must be aimed as * a completely new data area). */ - vma->vm_flags |= VM_SOFTDIRTY; + set_vm_flags(vma, VM_SOFTDIRTY); vma_set_page_prot(vma); @@ -2959,7 +2959,7 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, anon_vma_interval_tree_pre_update_vma(vma); } vma->vm_end = addr + len; - vma->vm_flags |= VM_SOFTDIRTY; + set_vm_flags(vma, VM_SOFTDIRTY); mas_store_prealloc(mas, vma); if (vma->anon_vma) { @@ -2979,7 +2979,7 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, vma->vm_start = addr; vma->vm_end = addr + len; vma->vm_pgoff = addr >> PAGE_SHIFT; - vma->vm_flags = flags; + init_vm_flags(vma, flags); vma->vm_page_prot = vm_get_page_prot(flags); mas_set_range(mas, vma->vm_start, addr + len - 1); if (mas_store_gfp(mas, vma, GFP_KERNEL)) @@ -2992,7 +2992,7 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, mm->data_vm += len >> PAGE_SHIFT; if (flags & VM_LOCKED) mm->locked_vm += (len >> PAGE_SHIFT); - vma->vm_flags |= VM_SOFTDIRTY; + set_vm_flags(vma, VM_SOFTDIRTY); validate_mm(mm); return 0; diff --git a/mm/mprotect.c b/mm/mprotect.c index 908df12caa26..79adae74c094 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -633,7 +633,7 @@ mprotect_fixup(struct mmu_gather *tlb, struct vm_area_struct *vma, * vm_flags and vm_page_prot are protected by the mmap_lock * held in write mode. */ - vma->vm_flags = newflags; + reset_vm_flags(vma, newflags); if (vma_wants_manual_pte_write_upgrade(vma)) mm_cp_flags |= MM_CP_TRY_CHANGE_WRITABLE; vma_set_page_prot(vma); diff --git a/mm/mremap.c b/mm/mremap.c index 5f6f9931bff1..2ccdd1561f5b 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -661,7 +661,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, /* Conceal VM_ACCOUNT so old reservation is not undone */ if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) { - vma->vm_flags &= ~VM_ACCOUNT; + clear_vm_flags(vma, VM_ACCOUNT); excess = vma->vm_end - vma->vm_start - old_len; if (old_addr > vma->vm_start && old_addr + old_len < vma->vm_end) @@ -716,9 +716,9 @@ static unsigned long move_vma(struct vm_area_struct *vma, /* Restore VM_ACCOUNT if one or two pieces of vma left */ if (excess) { - vma->vm_flags |= VM_ACCOUNT; + set_vm_flags(vma, VM_ACCOUNT); if (split) - find_vma(mm, vma->vm_end)->vm_flags |= VM_ACCOUNT; + set_vm_flags(find_vma(mm, vma->vm_end), VM_ACCOUNT); } return new_addr; diff --git a/mm/nommu.c b/mm/nommu.c index 214c70e1d059..b3154357ced5 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -173,7 +173,7 @@ static void *__vmalloc_user_flags(unsigned long size, gfp_t flags) mmap_write_lock(current->mm); vma = find_vma(current->mm, (unsigned long)ret); if (vma) - vma->vm_flags |= VM_USERMAP; + set_vm_flags(vma, VM_USERMAP); mmap_write_unlock(current->mm); } @@ -991,7 +991,8 @@ static int do_mmap_private(struct vm_area_struct *vma, atomic_long_add(total, &mmap_pages_allocated); - region->vm_flags = vma->vm_flags |= VM_MAPPED_COPY; + set_vm_flags(vma, VM_MAPPED_COPY); + region->vm_flags = vma->flags; region->vm_start = (unsigned long) base; region->vm_end = region->vm_start + len; region->vm_top = region->vm_start + (total << PAGE_SHIFT); @@ -1088,7 +1089,7 @@ unsigned long do_mmap(struct file *file, region->vm_flags = vm_flags; region->vm_pgoff = pgoff; - vma->vm_flags = vm_flags; + init_vm_flags(vma, vm_flags); vma->vm_pgoff = pgoff; if (file) { @@ -1152,7 +1153,7 @@ unsigned long do_mmap(struct file *file, vma->vm_end = start + len; if (pregion->vm_flags & VM_MAPPED_COPY) - vma->vm_flags |= VM_MAPPED_COPY; + set_vm_flags(vma, VM_MAPPED_COPY); else { ret = do_mmap_shared_file(vma); if (ret < 0) { @@ -1632,7 +1633,7 @@ int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, if (addr != (pfn << PAGE_SHIFT)) return -EINVAL; - vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP); return 0; } EXPORT_SYMBOL(remap_pfn_range); diff --git a/mm/secretmem.c b/mm/secretmem.c index 04c3ac9448a1..334b85714bd7 100644 --- a/mm/secretmem.c +++ b/mm/secretmem.c @@ -128,7 +128,7 @@ static int secretmem_mmap(struct file *file, struct vm_area_struct *vma) if (mlock_future_check(vma->vm_mm, vma->vm_flags | VM_LOCKED, len)) return -EAGAIN; - vma->vm_flags |= VM_LOCKED | VM_DONTDUMP; + set_vm_flags(vma, VM_LOCKED | VM_DONTDUMP); vma->vm_ops = &secretmem_vm_ops; return 0; diff --git a/mm/shmem.c b/mm/shmem.c index c301487be5fb..2096bbdc955f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2289,7 +2289,7 @@ static int shmem_mmap(struct file *file, struct vm_area_struct *vma) return ret; /* arm64 - allow memory tagging on RAM-based files */ - vma->vm_flags |= VM_MTE_ALLOWED; + set_vm_flags(vma, VM_MTE_ALLOWED); file_accessed(file); /* This is anonymous shared memory if it is unlinked at the time of mmap */ diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ca71de7c9d77..da02ec9c650f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3657,7 +3657,7 @@ int remap_vmalloc_range_partial(struct vm_area_struct *vma, unsigned long uaddr, size -= PAGE_SIZE; } while (size > 0); - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); return 0; } diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index c567d5e8053e..30158585c688 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -1890,10 +1890,10 @@ int tcp_mmap(struct file *file, struct socket *sock, { if (vma->vm_flags & (VM_WRITE | VM_EXEC)) return -EPERM; - vma->vm_flags &= ~(VM_MAYWRITE | VM_MAYEXEC); + clear_vm_flags(vma, VM_MAYWRITE | VM_MAYEXEC); /* Instruct vm_insert_page() to not mmap_read_lock(mm) */ - vma->vm_flags |= VM_MIXEDMAP; + set_vm_flags(vma, VM_MIXEDMAP); vma->vm_ops = &tcp_vm_ops; return 0; diff --git a/security/selinux/selinuxfs.c b/security/selinux/selinuxfs.c index 0a6894cdc54d..9037deb5979e 100644 --- a/security/selinux/selinuxfs.c +++ b/security/selinux/selinuxfs.c @@ -262,7 +262,7 @@ static int sel_mmap_handle_status(struct file *filp, if (vma->vm_flags & VM_WRITE) return -EPERM; /* disallow mprotect() turns it into writable */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); return remap_pfn_range(vma, vma->vm_start, page_to_pfn(status), @@ -506,13 +506,13 @@ static int sel_mmap_policy(struct file *filp, struct vm_area_struct *vma) { if (vma->vm_flags & VM_SHARED) { /* do not allow mprotect to make mapping writable */ - vma->vm_flags &= ~VM_MAYWRITE; + clear_vm_flags(vma, VM_MAYWRITE); if (vma->vm_flags & VM_WRITE) return -EACCES; } - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_ops = &sel_mmap_policy_ops; return 0; diff --git a/sound/core/oss/pcm_oss.c b/sound/core/oss/pcm_oss.c index ac2efeb63a39..52473e2acd07 100644 --- a/sound/core/oss/pcm_oss.c +++ b/sound/core/oss/pcm_oss.c @@ -2910,7 +2910,7 @@ static int snd_pcm_oss_mmap(struct file *file, struct vm_area_struct *area) } /* set VM_READ access as well to fix memset() routines that do reads before writes (to improve performance) */ - area->vm_flags |= VM_READ; + set_vm_flags(area, VM_READ); if (substream == NULL) return -ENXIO; runtime = substream->runtime; diff --git a/sound/core/pcm_native.c b/sound/core/pcm_native.c index 9c122e757efe..f716bdb70afe 100644 --- a/sound/core/pcm_native.c +++ b/sound/core/pcm_native.c @@ -3675,8 +3675,9 @@ static int snd_pcm_mmap_status(struct snd_pcm_substream *substream, struct file return -EINVAL; area->vm_ops = &snd_pcm_vm_ops_status; area->vm_private_data = substream; - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; - area->vm_flags &= ~(VM_WRITE | VM_MAYWRITE); + mod_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP, + VM_WRITE | VM_MAYWRITE); + return 0; } @@ -3712,7 +3713,7 @@ static int snd_pcm_mmap_control(struct snd_pcm_substream *substream, struct file return -EINVAL; area->vm_ops = &snd_pcm_vm_ops_control; area->vm_private_data = substream; - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP); return 0; } @@ -3828,7 +3829,7 @@ static const struct vm_operations_struct snd_pcm_vm_ops_data_fault = { int snd_pcm_lib_default_mmap(struct snd_pcm_substream *substream, struct vm_area_struct *area) { - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP); if (!substream->ops->page && !snd_dma_buffer_mmap(snd_pcm_get_dma_buf(substream), area)) return 0; diff --git a/sound/soc/pxa/mmp-sspa.c b/sound/soc/pxa/mmp-sspa.c index fb5a4390443f..fdd72d9bb46c 100644 --- a/sound/soc/pxa/mmp-sspa.c +++ b/sound/soc/pxa/mmp-sspa.c @@ -404,7 +404,7 @@ static int mmp_pcm_mmap(struct snd_soc_component *component, struct snd_pcm_substream *substream, struct vm_area_struct *vma) { - vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(vma, VM_DONTEXPAND | VM_DONTDUMP); vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); return remap_pfn_range(vma, vma->vm_start, substream->dma_buffer.addr >> PAGE_SHIFT, diff --git a/sound/usb/usx2y/us122l.c b/sound/usb/usx2y/us122l.c index e558931cce16..b51db622a69b 100644 --- a/sound/usb/usx2y/us122l.c +++ b/sound/usb/usx2y/us122l.c @@ -224,9 +224,9 @@ static int usb_stream_hwdep_mmap(struct snd_hwdep *hw, } area->vm_ops = &usb_stream_hwdep_vm_ops; - area->vm_flags |= VM_DONTDUMP; + set_vm_flags(area, VM_DONTDUMP); if (!read) - area->vm_flags |= VM_DONTEXPAND; + set_vm_flags(area, VM_DONTEXPAND); area->vm_private_data = us122l; atomic_inc(&us122l->mmap_count); out: diff --git a/sound/usb/usx2y/usX2Yhwdep.c b/sound/usb/usx2y/usX2Yhwdep.c index c29da0341bc5..3abe6d891f98 100644 --- a/sound/usb/usx2y/usX2Yhwdep.c +++ b/sound/usb/usx2y/usX2Yhwdep.c @@ -61,7 +61,7 @@ static int snd_us428ctls_mmap(struct snd_hwdep *hw, struct file *filp, struct vm } area->vm_ops = &us428ctls_vm_ops; - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP); area->vm_private_data = hw->private_data; return 0; } diff --git a/sound/usb/usx2y/usx2yhwdeppcm.c b/sound/usb/usx2y/usx2yhwdeppcm.c index 767a227d54da..22ce93b2fb24 100644 --- a/sound/usb/usx2y/usx2yhwdeppcm.c +++ b/sound/usb/usx2y/usx2yhwdeppcm.c @@ -706,7 +706,7 @@ static int snd_usx2y_hwdep_pcm_mmap(struct snd_hwdep *hw, struct file *filp, str return -ENODEV; area->vm_ops = &snd_usx2y_hwdep_pcm_vm_ops; - area->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP; + set_vm_flags(area, VM_DONTEXPAND | VM_DONTDUMP); area->vm_private_data = hw->private_data; return 0; } From patchwork Mon Jan 9 20:53:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4343CC61DB3 for ; Mon, 9 Jan 2023 20:54:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB6478E0003; Mon, 9 Jan 2023 15:54:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D66E08E0001; Mon, 9 Jan 2023 15:54:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE0548E0003; Mon, 9 Jan 2023 15:54:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B10FA8E0001 for ; Mon, 9 Jan 2023 15:54:24 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8674EAB12C for ; Mon, 9 Jan 2023 20:54:24 +0000 (UTC) X-FDA: 80336463648.03.23B3E59 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf19.hostedemail.com (Postfix) with ESMTP id E7B8F1A0009 for ; Mon, 9 Jan 2023 20:54:22 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kK6VGqQr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3_n68YwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_n68YwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297662; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=N7mRvZOR9OSs0kq6Wz0vbjqUGqop/n2M19/OmQJBzkI=; b=m7UE82BdMJ5EiobSC5HdCzFwrkBiowpEox0X5MOi8qVFErqy9bwx3CDM5Zy8eKMkHTRbNd rJXGeS4vU5GM7erLoN/k0VQj/tPewBwArkVYupbs+kDMNejeGadUAJZ0uMEcq8IJGkIhmF agW//qtPy76EGu2zIF19RjVS3zG9FUI= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kK6VGqQr; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of 3_n68YwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3_n68YwYKCBMBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297662; a=rsa-sha256; cv=none; b=GmwIQB1jNoxlL8pe6je2qR73WjGPBRH4OCLzco/nt+T49x9CcWPpDvt3u/RBkfRHy9943I aX7xGNN2qV4NtOQPzjpI3wBbFcTHvzOnxjGRJApLUKvAjw5MIIJnBEw6vhovy/IHNHHPi7 uNcDPbTU6bj8ccu56uIiB7e7dOuuFs8= Received: by mail-yb1-f201.google.com with SMTP id k204-20020a256fd5000000b007b8b040bc50so8773940ybc.1 for ; Mon, 09 Jan 2023 12:54:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=N7mRvZOR9OSs0kq6Wz0vbjqUGqop/n2M19/OmQJBzkI=; b=kK6VGqQrZF8SG+/SjIkBNYvs6AD0XP75PAW8W4oCooA8s3XRGaAuXa1jS1HIr83jUq PfI+R4O3x61YrRtYzJolldiJm8hvHC8SxT9psAXnN9HRUclq2Ip2r4U5z9IkvWwPkq1G R8EmUXB5Ci5VhDTGcvMKuR9ocC5TZKh6cUC/3KU3F2jjjEENLe5/WyQt1N1DazjgpxS4 yljAKeS+kCfpqMArc2lBS+96rZHSjRTZ1IN5B+phw/v3GaAeI+j/XkkH3A++TUlQdWQv Z4M4yuHxGDrl8gjpdpiZsATGjMq3MZe3RurXo4cwDj7NtnBXJ6X3cbtci2E7Lsi2JjQl 1UiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=N7mRvZOR9OSs0kq6Wz0vbjqUGqop/n2M19/OmQJBzkI=; b=N+VkiNRKySsp4A9AmMFWt0ge4d+6OfuxVDnyfsmyH3GinfXz0BlucuXNSbijImLLHi 7NK6V1TQsRIGrJX667XQDH1MeNCN772w7IPB5h1mfNYNk274tVzzUfe6vFMTszMUa8km V5mliu5SJ0hLSUZrfH4r60A3Ys3AS4VqS5qzxzulLu+4dAgBh5A52M2HO7+C3GNNR6IT lGYjDKsaDItfhJtQVSc0ODuHXYGfwqNjLTqtT2oBHOjwkLnwDwgSK1W5ef6oUYl0ZZ/G XI4NkFbN2KI5em70qNie9xFHq54DIm0pPjDdV9aWWvK1EZn0mD1Ej8yci3sZbNQBTozu YoZQ== X-Gm-Message-State: AFqh2kpfLJxV3z3EKH3Z6CtyuljnNk1//8fohzYJY8R+0ygtfwfVV/z+ pQyqubEGFtfliLU7Kx/neKbWyhLxuMQ= X-Google-Smtp-Source: AMrXdXshiyJrN6EY9CYZGhpA9uWntRCGsbXmXt4bSfqvBPsR2AoGwmZNzha8yCXqkiCEpIRfWXrumcAFYxY= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:65d7:0:b0:39a:afeb:f519 with SMTP id z206-20020a8165d7000000b0039aafebf519mr799450ywb.146.1673297662050; Mon, 09 Jan 2023 12:54:22 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:11 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-17-surenb@google.com> Subject: [PATCH 16/41] mm: replace vma->vm_flags indirect modification in ksm_madvise From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: E7B8F1A0009 X-Stat-Signature: bhy5c94eeugt86cnusdatre7at37otoa X-HE-Tag: 1673297662-325219 X-HE-Meta: U2FsdGVkX1++IY7iYFKafV/+iFkouqgmCwM47/4ga0FLBQ4DZVcZZO6qTgA8/22aPr/nr3KsXk8A1H6ODm80VqhgcKdiotILJhKHvn9ghxWNIEfqAKg4I1T5PtGNB46MsYrUfmQ3nydvvK7Bvun/gnavhPHCLKufeIKvkKScRng7HyEksFqYfZ8nAA6Uberxh6hhD4uXvUGxU2XxeMsgkWM0Gfg3SBbFhoSt6DUObSXhQW6RAKT3i3lCVvnysDjFUDXa43QxcfDPdaABWameco+Li1QSZKp74Z4YnXNTq+TkIvrbf2JFPw0o8XULcDlbHZsjjLAZVy8mnPWPV1Pj5EgepncclRi4lEp1femjoFszYla3lx4KiIT1hApsToeeMaIhFmOHG69O7wAfn9AStnF1DA8873BErDwheLH/W1bKGnVtlWYDMIjUd4HMdDRPjUW2yQtTvvTXt3o/ZN7P/5vaPmVKS1vsnjOdA2y/b3Ny8B8Aj9aXL0OCCmHLZ9IxyF0TcVh+Fsc8Vn9E+UnJnyyM/8mGI6ALqHZUwkt/NJZyli9Vfb5w6zPfWkAze4kgYYLDv0qD/Rc2qUqzxgBktuahzUvBtf3VjExksK1CYTvP+nFIU3H6YyQLCtyi5+of7fzSmXo+g86ycjiGlhvv/zvkt/CvdJSgzuWxd0ZbKvviwg+pkSkTqJ/i2bj3l/TNyfrjj6hFiNlHs/2kq1tZ0HB4Vht2ZeBvS3CVXdYfKij54QVm7OwHtYt5efLtF6g5axVHN0fbuP69ar97DpKHST4xbyRpJGkQupx7HT10ChpTcSJyIwczCVJnvpm63NosehM13jzfloEEoWI6UWHVdxPxsm9WJutFgqEO1Aw/GokKu5luq2VgEAjVXPzgvSqau/0t+l2Hk6Nq7D8VEgCKh/ChQrmytmGK5BnkiW2gSv7uXBoRmPBgoQc2Hu3PhqttDbifC815DFuj3gRCrPl 6bm/6mpj KJHIlFV6YM9blsyzP0nC5g0HDFSSe7YvIl+NtG9e8ZVV7RIwSQF1+14vdS7zm+0/VWNWiseengU2U4WWiSZontcPf7tnzIYiBmWG478zAhFjfoL05zapdfO2yu3ULWGq+5AzCJwEU23PrF44ujTUUuGTxJcN5rBmhWdztBgCk/AX3ogtKvQ2N41l5/9ETXYbkj/7tliKSkYlb98rY0pr9HnTlKlhW3ODndri9bYQuduk5tEwJ/V7VP2MA8X/WV2SohgqJE+wkpFhlwIFoTUEhXZKBatppfYsPV7wm7gcLUUCIMU/HFwbdEUiWwNTk5CQVPaQdX+lyYkt1OTKslF5tK3rJ42FbPhyOpYk/Qpp3CymddQVrpWzsfj0vkYruwZJ41o4goTcYPqNXNl9XkN7KOeJOtSJJJQRU8202 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace indirect modifications to vma->vm_flags with calls to modifier functions to be able to track flag changes and to keep vma locking correctness. Add a BUG_ON check in ksm_madvise() to catch indirect vm_flags modification attempts. Signed-off-by: Suren Baghdasaryan --- arch/powerpc/kvm/book3s_hv_uvmem.c | 5 ++++- arch/s390/mm/gmap.c | 5 ++++- mm/khugepaged.c | 2 ++ mm/ksm.c | 2 ++ 4 files changed, 12 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c index 1d67baa5557a..325a7a47d348 100644 --- a/arch/powerpc/kvm/book3s_hv_uvmem.c +++ b/arch/powerpc/kvm/book3s_hv_uvmem.c @@ -393,6 +393,7 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm, { unsigned long gfn = memslot->base_gfn; unsigned long end, start = gfn_to_hva(kvm, gfn); + unsigned long vm_flags; int ret = 0; struct vm_area_struct *vma; int merge_flag = (merge) ? MADV_MERGEABLE : MADV_UNMERGEABLE; @@ -409,12 +410,14 @@ static int kvmppc_memslot_page_merge(struct kvm *kvm, ret = H_STATE; break; } + vm_flags = vma->vm_flags; ret = ksm_madvise(vma, vma->vm_start, vma->vm_end, - merge_flag, &vma->vm_flags); + merge_flag, &vm_flags); if (ret) { ret = H_STATE; break; } + reset_vm_flags(vma, vm_flags); start = vma->vm_end; } while (end > vma->vm_end); diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 3811d6c86d09..e47387f8be6d 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2587,14 +2587,17 @@ int gmap_mark_unmergeable(void) { struct mm_struct *mm = current->mm; struct vm_area_struct *vma; + unsigned long vm_flags; int ret; VMA_ITERATOR(vmi, mm, 0); for_each_vma(vmi, vma) { + vm_flags = vma->vm_flags; ret = ksm_madvise(vma, vma->vm_start, vma->vm_end, - MADV_UNMERGEABLE, &vma->vm_flags); + MADV_UNMERGEABLE, &vm_flags); if (ret) return ret; + reset_vm_flags(vma, vm_flags); } mm->def_flags &= ~VM_MERGEABLE; return 0; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5cb401aa2b9d..5376246a3052 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -352,6 +352,8 @@ struct attribute_group khugepaged_attr_group = { int hugepage_madvise(struct vm_area_struct *vma, unsigned long *vm_flags, int advice) { + /* vma->vm_flags can be changed only using modifier functions */ + BUG_ON(vm_flags == &vma->vm_flags); switch (advice) { case MADV_HUGEPAGE: #ifdef CONFIG_S390 diff --git a/mm/ksm.c b/mm/ksm.c index dd02780c387f..d05c41b289db 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -2471,6 +2471,8 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start, struct mm_struct *mm = vma->vm_mm; int err; + /* vma->vm_flags can be changed only using modifier functions */ + BUG_ON(vm_flags == &vma->vm_flags); switch (advice) { case MADV_MERGEABLE: /* From patchwork Mon Jan 9 20:53:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19AFDC54EBD for ; Mon, 9 Jan 2023 20:54:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA9AB8E0005; Mon, 9 Jan 2023 15:54:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A59668E0001; Mon, 9 Jan 2023 15:54:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FA698E0005; Mon, 9 Jan 2023 15:54:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 814748E0001 for ; Mon, 9 Jan 2023 15:54:27 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5F16BA0B20 for ; Mon, 9 Jan 2023 20:54:27 +0000 (UTC) X-FDA: 80336463774.15.647C516 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf23.hostedemail.com (Postfix) with ESMTP id BFC1214000F for ; Mon, 9 Jan 2023 20:54:25 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oTTbEHXX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3AH-8YwYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3AH-8YwYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297665; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hDqG76VNjmKFCkYaVQCHk1jK9PV7CZBX+MdyRpSUw7Y=; b=KzD9UR8TzynNCOZ4eCA1Gnf+ITCLdnGE8kCYCvo3iCfOHxkTdE1kY9jq64S9Mm6874E72d ppYTejuKtxrw/s6dU9goEt7BwNED/5toMyMaKsf2eEW+a0xXlhmxJFwE2O7pJ1SvdCtm+P HW1hEtms9aUpxuQiSRdmgcpkNO07Z74= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oTTbEHXX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3AH-8YwYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3AH-8YwYKCBUDFCz8w19916z.x97638FI-775Gvx5.9C1@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297665; a=rsa-sha256; cv=none; b=a9HDxrU1G7DSl1lA75XSKLET6AC9CJOQQjVcKHVYrontY/AAlFZOVBBV9xT+N8l83tK5Pz WWi0twx6V795YX0GSSVqNZ9/xgzaGRV1xmeetCCrxbxTgZDsTw2g3RNhfHcI6AEeDZxS6I dKqeEUF9YW9T4f4dGep8fYMvBECyWxI= Received: by mail-pl1-f201.google.com with SMTP id l17-20020a170902f69100b001928d6b3efcso6963229plg.2 for ; Mon, 09 Jan 2023 12:54:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hDqG76VNjmKFCkYaVQCHk1jK9PV7CZBX+MdyRpSUw7Y=; b=oTTbEHXXf074UTT/LXfKIzTlYAOVSZBWjz/z2pi8uw2sgTsKOn0pf4M+FUmb3fhKEp t8utYb+BYUrtOmcMbBRfjQmj3HjZCnvt7nX8XXsh4x467hUw4mc17N9vBA4v3IH0gXYu Tf0NgJ8TRnq+dnC575YF2XUlugz8igLSTMm592zbLFSlzWIJOjCEMLUlmrt7z2QrvV2T gekww8A2UJVHgfp1zwqjj/KBiOxEMFrYzziSkpJrGHAD1djB4N2zU8kmwkE7PnEOrNUu hxvDlopUQbh5t04B3B6cW3kLOVcLSOY+TAkcMQBqHoc5+w1BV2sS39QIbB6FPKvgrwld s2sA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hDqG76VNjmKFCkYaVQCHk1jK9PV7CZBX+MdyRpSUw7Y=; b=QuRxgbpZkqDad8j123+2qfdoO1MUMtECwUTAAfb2CWqIE6oU5UGgx6gYUBsX4fKa2H XTigdAyBCMAqV7r7vrEWSmpY2rnmwt18uX0ZxNt1nLfFVlHM+nubjc1fhhQEhshYbUPh NKQ0OYWHlEuAFL9whKOhUWwNHV5+rKWo0WEufSb27dFZPnyGpk8CsVq4eC8hWKmdIUS5 FZKncEqIdqgmkyZrPqPfZ3IKJWPHw9j5vv6WSoUxAaxXNDKBijj8+E5VvMsWvg1UneY9 0JfiU4w71GC+t/coUGjNNW03WrBjI1lM0Mh5KrJNK1Ox+Ga8pA/5ni4ydcaUhcwINt/U O8Kg== X-Gm-Message-State: AFqh2ko+1Y+FMjUwH6wXY4DUxt/6B2RrW1Dqzb/F/HpvA/Gaf9EZm2Tl 70mo8PCUBn1PR5fM3OHPI568m+Ljyno= X-Google-Smtp-Source: AMrXdXvx+5mjUKCd2siyFSRbOdBiHgQPNh6ZnBtsj8lXWONeXJTph7ewao1odYnxoExvO1xzjoIbjT4zuzo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:90a:638b:b0:221:52e3:1f56 with SMTP id f11-20020a17090a638b00b0022152e31f56mr5166916pjj.225.1673297664681; Mon, 09 Jan 2023 12:54:24 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:12 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-18-surenb@google.com> Subject: [PATCH 17/41] mm/mmap: move VMA locking before anon_vma_lock_write call From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: BFC1214000F X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: ycf6u5eb4sstdk8dp74wrhyakqb4q9an X-HE-Tag: 1673297665-981271 X-HE-Meta: U2FsdGVkX185fx3L0EEl51sss+RBg3PZJ0Mmk0sgGVZ5g0w4rG6Ld7CXgOwdXpHEcgUmWD8xGyVTlIx6yQklFBXeukJq0p3aqUuRCm4W2chVF6nJfBbQtJWYpEkEgTWl2eiP8CBqJ7sP6PcLXzaUfOVXKrqkMdcb5abFpDHI/0hGXhjkndDRkqCQ6oNeSZRv8FDQshCK+4/o/qYgtxurQQPR2GI5zgDx13LL/KOViPZwuaHToGIYans7equoOP28akKiF17vCllLMoc0N8iIhyiPvwChfRJMi8zPDvvyVo3uGZa4PPZZnnOqVUZ/aa3YN/JEjBMUBy0QkjyzEVIB4ZL4bwVF601c+gzvuoEVROO0WRQNosDh+PCaY2a3z+WujjUaYneRsnVO7/8kDO3VmjzCQLh/GuTP3JP1pb+//o0RpomhbBXBJDN/s6aNZVAHGOIbuh2CyTVC+zM7o1qq7OFbdXXHK9iMLo76Bq5Wsgg+eFC+ptoYjwdjzYQq3OpYdlXngwMty90tGk9KiYd0J+cH1Is0PcszsbCrg8onVrRjO/CCpeUII1s0t7VcvbcKTF2phiE1+gkJE+p+W4Jy1DtNBaax5uC9FrnMhevecIDXLWT385GltnLvMxgPjHswA2o8Qu4JIcsdOHrjDVybGre47iNCjzgw2MX47m77Kt8G9nxqfC+drpbpUrnb1WO4rLwokG9PT3BqkLgnpUof3qSp7SAjyrgTc6HF9Xoyhwx/OiVsB+eztmrxBNheoBUByPWhQZvqsdglPZMXr7o/jCYCr32hFGzU/dOGwfX+oCBOxr7ZON7VZyT0t0HJsGGRpacU4WTPBk/laUENVJPZefnUPckzyZqdqQD2DQhNc9mRHOagkZt+iq7S1gzZJvy6xHrtZ9BEIe6Op9Uy9+Vm/fuq6u72s7aL4GNRagob/9hpdutxXAnESehjaaqa1/uzHFza44GM+xSOF+dcIwT 5/Dth6Xj LpacRn4Vn9GUlpow4izqmbIe7fd8NDcJ2W4Dm5BcwAK96UbgSwT2XFKsUP4Mbn9mUPov3Xx4h8tB5SRai1IrRsOxyO3nkwOtk2q2haZ1D8al0wKXMFhkX5lYfo8nxMb97lSPkHZnLu3i2oQ4gOM0YzS9lA4rxnMASbBKOF0Lxr1vlYjydtsPPyxoOj9a42sQHK43f/bu/ejwXDdMrG2dqyDviObwRg7+lPkD6J7U6MSzF6qRmVREtCAKJXaFimVHzfSAHSHhQbh/Mp8IgO5+1tCGmfzvcniKUUcs5YbtcHoAoqmb5kyJecpOohV71wZ/TTkJxVLlsGkobNBYqql9byK89ChVgNAtXIDpVkYx20dqvaZ+rE+1b4BJidI7fARmjp1ciguC+tYYUvVI9XSKwVCEynlS2JmjshjwK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Move VMA flag modification (which now implies VMA locking) before anon_vma_lock_write to match the locking order of page fault handler. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index fa994ae903d9..53d885e70a54 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2953,13 +2953,13 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, if (mas_preallocate(mas, vma, GFP_KERNEL)) goto unacct_fail; + set_vm_flags(vma, VM_SOFTDIRTY); vma_adjust_trans_huge(vma, vma->vm_start, addr + len, 0); if (vma->anon_vma) { anon_vma_lock_write(vma->anon_vma); anon_vma_interval_tree_pre_update_vma(vma); } vma->vm_end = addr + len; - set_vm_flags(vma, VM_SOFTDIRTY); mas_store_prealloc(mas, vma); if (vma->anon_vma) { From patchwork Mon Jan 9 20:53:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094284 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93DCEC6379F for ; Mon, 9 Jan 2023 20:54:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32F7F8E0014; Mon, 9 Jan 2023 15:54:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2DFC08E0001; Mon, 9 Jan 2023 15:54:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1803E8E0014; Mon, 9 Jan 2023 15:54:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 086DE8E0001 for ; Mon, 9 Jan 2023 15:54:30 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D90F21A0F57 for ; Mon, 9 Jan 2023 20:54:29 +0000 (UTC) X-FDA: 80336463858.28.E7E66CE Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 4A4408000B for ; Mon, 9 Jan 2023 20:54:28 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EbLgVpiN; spf=pass (imf30.hostedemail.com: domain of 3A3-8YwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3A3-8YwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297668; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mttq8CcSxh6pk/PDvvqo/9XfYX/awSV+YKJHUec4DWQ=; b=3f/hR/CSqDQ3F9duaDtLOuC+dF1gXPMtJYpUWXv7jIILq9dWbcieBYUdf2KsMLn5vrpasZ w+DWiqHWVSqyzwwy+qblmkR50HipisnyrY0icoa5dRTTKiVMMkFYkLi1AHLUhwLy2FwXf4 +e8V4YMLn29tXbS8z7Ndg0at0jqYaW8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EbLgVpiN; spf=pass (imf30.hostedemail.com: domain of 3A3-8YwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3A3-8YwYKCBgGIF2Bz4CC492.0CA96BIL-AA8Jy08.CF4@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297668; a=rsa-sha256; cv=none; b=Qnfiy/Q1IjMbNhzE8HW65LXi8J7W3Fp4BsAEGjKt0kq3eq68fLeAftDIhecUAe7MpY6UF7 SyDdm1nzE5jfVMjv2a5RKDaer1FcTutHT7IyMfweWVYSOUKLXbgQTUa1ASxkwW+o0jTNha 2DXGHp4p1sRfrPKvWhmTdfV8jJsyBfc= Received: by mail-yb1-f201.google.com with SMTP id i10-20020a25f20a000000b006ea4f43c0ddso10381013ybe.21 for ; Mon, 09 Jan 2023 12:54:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mttq8CcSxh6pk/PDvvqo/9XfYX/awSV+YKJHUec4DWQ=; b=EbLgVpiNWauCxYsPhZzluNh8i7Jc9od95XU7q4baD/bZuTFbACugKjaBqwn3ZlwjPK DyA2dJROokooe1X9hWF9cqxx7R8dzaLC4TQ2VgG1K6ZzCrV6Cn+bNgWtTm8qKV2EPlQI C3EgQy0lx8995KoeqPGehRxKjsY9DJj6pO/ydpCP34y4VCDsA5lFDA9xaQObsm54nagL gf/rqX49ePSI64VKn0w7yUB28hwRSupVRZyvn8G3inMqKMvhkMvaFS5iHqspfhqeeQGW sJQNxcx4h1OW/3tP5B8wZxgrbXFTXvC0zscvDZLigjE3sBlwiKuCEIrG4zHiBAmIFXhN sp1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mttq8CcSxh6pk/PDvvqo/9XfYX/awSV+YKJHUec4DWQ=; b=Tql8ruPCZqb+8myY2zYuLkZ7yB+i6YqYJIpzUPvumkA1SOq/M+/wQctT0zNeXoLNM2 /SHRnhf9XWPOEf8Sq/xL5ZgCVf0JOobMTItIMOeX+/8cBfz3DGtAeOGn869ibsS7h+oy 3Vy6V5ixr1LlrbQL0XRFv5NQzpe7JhnnXeASDTvl1+pTwCrGdTkOXrH0qGsxjIwMsNOu 2CyvC/VvUQgcKsLq1BSLrP6FNuab5XnMcW23GEECKvZy13S1xqWugxqZaftzuOO5YA+o U8MKozQV/AyA9zoIEXKX4BlWfoUIDLkknMoL+hqHhuADQMGxxVGZMlIbXUFd9V32d4f6 K06g== X-Gm-Message-State: AFqh2kpoYlxXnQRvRBcHENSFuLzzrJ665npOnkyZMHGy9dgz5vS/bQ1w 4ricui9i3g2wkhgMBq7UMnZn/lSgm9c= X-Google-Smtp-Source: AMrXdXsB8fbMiTP1DYP5J5XEU88bTBTK0YYISNp2DJzOoqciXF/+cmYxWMxqmdOzbjRBqH/+xJjk8vQehVo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:850e:0:b0:6f8:42d8:2507 with SMTP id w14-20020a25850e000000b006f842d82507mr8097864ybk.110.1673297667509; Mon, 09 Jan 2023 12:54:27 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:13 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-19-surenb@google.com> Subject: [PATCH 18/41] mm/khugepaged: write-lock VMA while collapsing a huge page From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4A4408000B X-Stat-Signature: 88komr9ghq87nocbfowf6fwewkd14kz8 X-HE-Tag: 1673297668-985419 X-HE-Meta: U2FsdGVkX18lI/xHcOlkXg6qJOFd5gxZaDIoEU9PBuj8Riw+bgS9X9x7f3eRc9GQXusK9V/kaddGdORAwVYbgvpy73ZlKz5TyHbk2oHL7zDxASjlPgxuzcx2GIZ0oM8NimJy3V7VWj/4/AZ0OhwSqsmX0xGg7PE7bXf7ch7MS7SdrSHln825gsUBompaWYTGpCldRfvIgpaojEtg5ojK0i52Pu1/sybzr2pNzFOe3uI8Xh25ouLiVhNMN1sbJtPVl44Ft8475Y0z5t917kQGHVFTUtUbKFxmEbfeySaMfmPzt9EO25JFsnEY3bztTV8F84sRzPBq41ETHGInLGu8cZvvsIps2mdfoMKYpHrkpAfXis6hyBsCMdDjpDihNZi88pGmVsO/oXg3TlaxoAqU/SlzFCmyTNEQH6sBtc0uyscAlw2gW0AIvsaK72sxXeLo96hF9T+UwnkI8b8X72GF0e6oSi3Dhr59SmzGigtANflQOvaNpitvV68Qa1aaZpUgaidPTfLF0HdQaSMUg0Yy78j6ssE7PBA25nGiEdU5xxYCN9UKFjLcrqObzK4tRButCsj4nB69Cxnva+qRFcaOTB0lp2vkSXo4UuyOHeWTjbKwZleMYlCd7b9pNwZEQgA+ydFQkSeAACxZec5kQWpjvCew+9o3kQzq1HIWdWtdyPwrM5IDEbDTObHAPtbDOtkOSUTwzPTpHw/U++kNxzejr1DK47hK4jqPjj8piAPnV4UrF+BjLO8bKDuG1BtiJ6loPpXMz+I9QjpvRZGzDYCwQfJ1dHSOPSHQOjkgdKgKUAQoMdyooWvVTxE228tci5U1iJd172lkCtOpVXdFQR0iVj9pBOE/qM/2eMYVLEdVWyoyyi8nkV7ZpbJ79vEMTRuUoet9ixqiNQfRROb/GeSesBOUArezaoluQxCiKAAnONXPleNhW2A2+QVmKm3R191vD/0rWWqslbqBqZGF6Sg VfvQtiCS 6wqkftZXbFkavnJXj5w5fBbu5Zd1v5dc1X/MtbJKF+1wtRx3GTumWQTPNEhPK3+TR8upoVo0vyAkWx70hQN54L/uSl3UgI4iKcZbVPFBx1e6t/N7THKtBetfWCWwHIi3xjua8L/YZxHO8QSPGy7Q0X7ZJR+emsx1NTaL+6VYJTLC6TukJKOmotqZuH6DBRUmMfaab3KwjyAMXlxrg5wdFl4Qs50e+QzOlvK0Gs0ulty53Kr+clQ6qQY2yKgACKkE+ZjEYH+M3FOoAnlxNvgt+VThIuy6b4/ntxrcP5V+opkiEvxV64aGAtaFt4KzkppEBJ2uWIPAKxM5mVx+OU4887aYrY78GcpQGsC831SKvnQCePfk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Protect VMA from concurrent page fault handler while collapsing a huge page. Page fault handler needs a stable PMD to use PTL and relies on per-VMA lock to prevent concurrent PMD changes. pmdp_collapse_flush(), set_huge_pmd() and collapse_and_free_pmd() can modify a PMD, which will not be detected by a page fault handler without proper locking. Signed-off-by: Suren Baghdasaryan --- mm/khugepaged.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 5376246a3052..d8d0647f0c2c 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1032,6 +1032,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (result != SCAN_SUCCEED) goto out_up_write; + vma_write_lock(vma); anon_vma_lock_write(vma->anon_vma); mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, @@ -1503,6 +1504,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* Lock the vma before taking i_mmap and page table locks */ + vma_write_lock(vma); + /* * We need to lock the mapping so that from here on, only GUP-fast and * hardware page walks can access the parts of the page tables that @@ -1690,6 +1694,7 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, result = SCAN_PTE_UFFD_WP; goto unlock_next; } + vma_write_lock(vma); collapse_and_free_pmd(mm, vma, addr, pmd); if (!cc->is_khugepaged && is_target) result = set_huge_pmd(vma, addr, pmd, hpage); From patchwork Mon Jan 9 20:53:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB5EEC5479D for ; Mon, 9 Jan 2023 20:54:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C3EB8E0007; Mon, 9 Jan 2023 15:54:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 467598E0001; Mon, 9 Jan 2023 15:54:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26AF78E0007; Mon, 9 Jan 2023 15:54:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 17AA88E0001 for ; Mon, 9 Jan 2023 15:54:32 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E46CC40AC1 for ; Mon, 9 Jan 2023 20:54:31 +0000 (UTC) X-FDA: 80336463942.13.5B5E822 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf20.hostedemail.com (Postfix) with ESMTP id 4C7DF1C0013 for ; Mon, 9 Jan 2023 20:54:30 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MKIFYUVg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3BX-8YwYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3BX-8YwYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297670; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ves4Zx76RBv25M18zzcj7gVloTdTuNMdyKmYC5DDvz0=; b=aTu0+aASxhZ9p1Yhd293cZU7fxENdg2wkKyLmcRHNvJO43bdDgeabj81TD7aAqRSCfiL5P EURfh836DtefVcIEM520ISC9tEH7K8FlFklhm6LThkLzOQ/TYkewRnorl0i9iwj/Xrz3+s dK+sEfPS1UZ6mTBZNs9ZtsI8EVd5yWI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MKIFYUVg; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3BX-8YwYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3BX-8YwYKCBoIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297670; a=rsa-sha256; cv=none; b=eyurPjOnpHEM0NIVJIApVs0nT38+mZG0rfxxElnH4Ta5MYMrPtidfrvAD1JP2HD/kFApya XlFqrj5Zlb5q644YouV7sV/6RhuqJtueC5QlWpY6cGhjZcdebfKoGK0MrS6eFFxQ6we9Rt IoElPvKr8ovBqkKTwOvzoJE+iAKevdg= Received: by mail-yb1-f202.google.com with SMTP id r8-20020a252b08000000b007b989d5e105so6927160ybr.11 for ; Mon, 09 Jan 2023 12:54:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ves4Zx76RBv25M18zzcj7gVloTdTuNMdyKmYC5DDvz0=; b=MKIFYUVg3C+KcUmk/DiJLvwQZvDLVNMlr8DQ0MCXFdkv6hN2eBfUWDzxIZ7BfVHp8I 9ZQKQbnc3cjtIsfRVDcXo71Ba38rTMbcUdq5jPPQ/O/qUYTP0dyF6taFqjbQZ9FRkBHu 4AhRyAaM/Xs1rNZHH4s644beTjAgsiLfyufSEIgttXRExgdKqL3ufxpDIm4mlvORyhtg 8KPqhXKz4czDblD/OO3yCfwh5TQTP8fNUj9/04RnIZEyuuEXUuTlBqAldxaXV0tj/mD/ bYzPmov2AWSWqL0UrgwZuK2hqucg36ElYTON5SUI9h/73LoxJPTGa7ERIj/8Aw4bHycM nzHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ves4Zx76RBv25M18zzcj7gVloTdTuNMdyKmYC5DDvz0=; b=Dksm1Spuz7SDoRG+LJnSLk0koexIM6O8RF4NeIBZj6lFTbL4usZGf/5Jk0VIjWppvm sts0hZN9fmR+SayqMuIzbe7qElohziRsGC2F+GU9IkJZ00/yuRkrL/wsnU7xCKX4i8n4 vtD/SimJQxHvpSYfU+snCo3mAPZt3tNQdpMXoTmNZu68Pbs3BtPFitIQp3UhZlQPndyZ h/NZYfwwdN7m/BpYtA0hG59IvpAWMINdDbMS3rkE465IkmTw5FXrOPB/dno01bmJXdLd p2u/KwiShXYc53Xc7X2Rtsl1kTN+e6DbzWEnOwEsfgxhBCIGnF8xaGSgMKroIKoS0gSQ ZpZw== X-Gm-Message-State: AFqh2kpfWWyScKkd/aHhOHWb/2k/ByVnx3FnEZY6dFThwu0/M45bN9hK l3MnVE5He42FhmMV4hdAFRxfZSheIbA= X-Google-Smtp-Source: AMrXdXvMc8uw91kFZQ5TZdRrbkHWe3EKuKErf5QuBh5uD1fMXXAng4XpjPLPHBE1E+VnZgCLzgnV8hcsb8Y= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:a013:0:b0:4a9:884a:20c4 with SMTP id x19-20020a81a013000000b004a9884a20c4mr4780346ywg.139.1673297669408; Mon, 09 Jan 2023 12:54:29 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:14 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-20-surenb@google.com> Subject: [PATCH 19/41] mm/mmap: write-lock VMAs before merging, splitting or expanding them From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 4C7DF1C0013 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 4o39mepnhqzpbroixp6ebxcaggp3rea1 X-HE-Tag: 1673297670-629943 X-HE-Meta: U2FsdGVkX1/zTlc6j2CClufGyiI5xPAVk/KMOuUjJknt7ZdAUMZHgWy30/JYVNP61cNCsjijjnPKG+18TThdj6Zl9uAUfCHLxcolaUH9H+HzBZjBmG3/4Vhm4lbWQP+fv1NI645xKDeFo7hkjtuRbpHuFO1J9+p1CEXL+Fw6mU5cADPX4Vifr8PNDNSfFpQF1qqmd+VAHV0GrFVFtWTrGhhXNvYV3eOb4e2669wj3LX45xULs6KCStZ6Fa5BvCEgHH0foz1JFtZexQm6OILhZ2jFGzJd56Zi/BeMRQr/xdOLD68ubZ03euZMpySZ0ap8WRA0B7nygSDkuBdhiqKXbHrHsZqFk45L1UrstPxCH3Tl45JpaIFSKz0fnYXg3GtJIyMecoA9rlUPTibEzbQrqa5IwCU9tvZAMlnJZQ5LnEHOtvtq7ms/f5Y9l5+pIJot/WjszhH2na/muGCZC1193UWo5jnG9i0CRu9qVCSm5ra3HItTtKKfUYPvvW2prsPTK1lbSXUFbpslBiduD1C+b53Vjhs3Tu20gjWSdjrq8iWSygUN6hFp7p5DXaPSB77AGuZr6fpWBH/Ub9dBOM3SqxghGGDSJa6OCeh7u7SCvp/sWha9vYV5dG3u9OgHivY9bBbZvocNUyUcGAaAOd51H3gEObOeSK0xFBoMw00Uu5RqGXBqPjTi32ePvylWHNS32t2srMy9tY8HGtflUSz7awAhN3b8iv7aQPchFLNdnLmBnS11hcyKD6OqmwWig+7t7vFpU6dDzwwgs2EuxkWG+b+HzmCuhaFfbn/ufbJzR2CA5wHoZaXt7JRaerVHSD7nFwAKM0xSN+3zYfdShG8waK2BoxyKesRS3fUsPluzs8hs4Qc1rRxbLqtjTu6+1nu+JcI3oRlAQpAQNGXRkReb8h+BVNO6BehyLzBWxbOD9IxeMnqrmu3pOaYv5fD5yOHH20PtFS8amnRH8IRZSAu xKziDLeJ CPtH8s7EE/tMPUeCeaGTvKufHBF5uQTYGQSgijFTVRvkeQr2Jiu+3TdKMCrqB/mSZ2VZANLNxG1MUVpTg6BipO1AVxAh1vDbMx/QG3Q/2SPxoXoVYtNzqmYDpUNEsP8uuONGHFY9NMkJAXVSdA/b3Lq03lwoyGey+NAJzLD5aITIRaaMDJBiKeZ8uICmfCqZpfTrQktp0dlzCzcyDk51GRO99GZgOrcyxV6+SdYHDARfkh9O3uuKn2u3RA09Y1DPLhhgWKvoL7lhIz+BIwx+SsnVdi1d2Om0Bd0zF80zng0OjztCD6HwSiBIjlthOUrqdj9Ib5voVzDrE0YQHVRmfwunbkDUu4BiYyB1Mg6DrXz0kzHE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Decisions about whether VMAs can be merged, split or expanded must be made while VMAs are protected from the changes which can affect that decision. For example, merge_vma uses vma->anon_vma in its decision whether the VMA can be merged. Meanwhile, page fault handler changes vma->anon_vma during COW operation. Write-lock all VMAs which might be affected by a merge or split operation before making decision how such operations should be performed. Not sure if expansion really needs this, just being paranoid. Otherwise mmap_region and vm_brk_flags might not locking. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 53d885e70a54..f6ca4a87f9e2 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -254,8 +254,11 @@ SYSCALL_DEFINE1(brk, unsigned long, brk) */ mas_set(&mas, oldbrk); next = mas_find(&mas, newbrk - 1 + PAGE_SIZE + stack_guard_gap); - if (next && newbrk + PAGE_SIZE > vm_start_gap(next)) - goto out; + if (next) { + vma_write_lock(next); + if (newbrk + PAGE_SIZE > vm_start_gap(next)) + goto out; + } brkvma = mas_prev(&mas, mm->start_brk); /* Ok, looks good - let it rip. */ @@ -1017,10 +1020,17 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm, if (vm_flags & VM_SPECIAL) return NULL; + if (prev) + vma_write_lock(prev); next = find_vma(mm, prev ? prev->vm_end : 0); mid = next; - if (next && next->vm_end == end) /* cases 6, 7, 8 */ + if (next) + vma_write_lock(next); + if (next && next->vm_end == end) { /* cases 6, 7, 8 */ next = find_vma(mm, next->vm_end); + if (next) + vma_write_lock(next); + } /* verify some invariant that must be enforced by the caller */ VM_WARN_ON(prev && addr <= prev->vm_start); @@ -2198,6 +2208,7 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma, int err; validate_mm_mt(mm); + vma_write_lock(vma); if (vma->vm_ops && vma->vm_ops->may_split) { err = vma->vm_ops->may_split(vma, addr); if (err) @@ -2564,6 +2575,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, /* Attempt to expand an old mapping */ /* Check next */ + if (next) + vma_write_lock(next); if (next && next->vm_start == end && !vma_policy(next) && can_vma_merge_before(next, vm_flags, NULL, file, pgoff+pglen, NULL_VM_UFFD_CTX, NULL)) { @@ -2573,6 +2586,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, } /* Check prev */ + if (prev) + vma_write_lock(prev); if (prev && prev->vm_end == addr && !vma_policy(prev) && (vma ? can_vma_merge_after(prev, vm_flags, vma->anon_vma, file, pgoff, vma->vm_userfaultfd_ctx, NULL) : @@ -2942,6 +2957,8 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) return -ENOMEM; + if (vma) + vma_write_lock(vma); /* * Expand the existing vma if possible; Note that singular lists do not * occur after forking, so the expand will only happen on new VMAs. From patchwork Mon Jan 9 20:53:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01978C54EBD for ; Mon, 9 Jan 2023 20:54:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D4398E0006; Mon, 9 Jan 2023 15:54:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 85E938E0001; Mon, 9 Jan 2023 15:54:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D7928E0006; Mon, 9 Jan 2023 15:54:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5FA8E8E0001 for ; Mon, 9 Jan 2023 15:54:34 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 40DA3AB041 for ; Mon, 9 Jan 2023 20:54:34 +0000 (UTC) X-FDA: 80336464068.29.7602738 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id A37E4160015 for ; Mon, 9 Jan 2023 20:54:32 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=m9qNPQfv; spf=pass (imf08.hostedemail.com: domain of 3B3-8YwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3B3-8YwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297672; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FDedOk4BDhI0jXVeiXMnt29MYEOAHvgEHolCeEtDVh8=; b=3lh1CWmHkR3shamHlUsvKB6jbMX4E2snQv9E3leLsH+weQEQFmhWhO+BN9YLX/mbKRdEkR RaYMZWkXS16/fmx9uWBaHW1u5KH7xF3FJ+hbTEw6fbq31Fzpi5pviVuA9GksHVcyfYk2NC k1W7L33LtT6fNrtvsV2vOPfJsEGv82Y= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=m9qNPQfv; spf=pass (imf08.hostedemail.com: domain of 3B3-8YwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3B3-8YwYKCBwKMJ6F38GG8D6.4GEDAFMP-EECN24C.GJ8@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297672; a=rsa-sha256; cv=none; b=Z3BOCkog/y7BUXlKbhum52M94xe0sZTt0z9hrl9eMHEPTRjWISouNkWpJ3NCjO1vn/X2vP L1QSy9IrlyYHGRW5pmNvN4Eff40SOeu5ZdMlzUlsRuQgRhCftcS7QH+eBAWQArO1hewf6E dxWDL2kx0uXDow3IPP52pz9RyOuG43g= Received: by mail-yb1-f201.google.com with SMTP id i17-20020a25bc11000000b007b59a5b74aaso10157464ybh.7 for ; Mon, 09 Jan 2023 12:54:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=FDedOk4BDhI0jXVeiXMnt29MYEOAHvgEHolCeEtDVh8=; b=m9qNPQfvAaSk/cZYvrILeQolkFuo8oX0rPOdz+ZeBZe0viaaMWSs2xM9wojEerSzR9 Ehc0liLdN4Rf9t9ITHbgyUo/TpjLKodzNOt3cYyWGhisrg0tQj8O7DOpS0TtK3tPQLQG qUB/3EWr0qIxJ9v9dQ2Qpf6kyj6P77wOGaZ+XxUB3NlRJ4nvQo0f4D36W3MPMO4ktFMc ysLVV5w6Mp+QnxSVtHOQDDwbxZ7kpIUBpnDdTFd6J6q0MuwMTyvve19XzRy6+EFEsY7p 5CEU/qiYVWTESVV3KMWaGXvUZnEBFFh/Tdv+49ubumcalCQjcVJZu4o8Eo8hhtY4lcfd RRGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=FDedOk4BDhI0jXVeiXMnt29MYEOAHvgEHolCeEtDVh8=; b=buFbuiDlpuxfJHauCOGuUMRNyMiczx9BAN9MkOYq7Du4qPY/MoK7k4SMdBuwaDmE66 8hxeNkW5RFnUZl7y73320rYxNHOgekXD55ZW0osixvHqSx/lGipMHt4CACHYZes278TB e7R/5lLUYp+jv5Jt3/5x09GegJEJIZnjveUPhqNb/fgTLAyhH7aSNBaCrVyDIF8g90XM v5YNRvEvtwXOp/anoX4d6LrNz4NpA2C4S1qXbuMYr3hqYCdcBV/AOrD2I4w5CS4IoHhs VdumsxTvhYEF3iz1DYcf8RgnBp/G8FtZ5HX5my2aY9z/76uA6q4yQ4rgtbdTEkZAxwc7 z65Q== X-Gm-Message-State: AFqh2kqyj8Fdu4uz8VHcq1+eMz0ioS44bK2lR15Lkck/G3EtoH+Heh5x uAo18iEiwM352PKsba6Hfdq+JK8OctY= X-Google-Smtp-Source: AMrXdXvGyOxiOv+MuSWvovzrd0so3mEpN1Edvcx+Z2HBXoeLvz+P6Hw4SXaxw2akuo9mk6+zHwkliVPrxQU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:1489:0:b0:6fb:ef1e:bb1a with SMTP id 131-20020a251489000000b006fbef1ebb1amr6925224ybu.168.1673297671808; Mon, 09 Jan 2023 12:54:31 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:15 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-21-surenb@google.com> Subject: [PATCH 20/41] mm/mmap: write-lock VMAs in vma_adjust From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: A37E4160015 X-Stat-Signature: mghsnz1y3bon1efrufa167o1cx6iaebs X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673297672-252037 X-HE-Meta: U2FsdGVkX19lHs8pqK17VfJVplkow/RvROGUFBEQhXZsURklACpNdalscCt9B7U6PUKmumoxVGugVxcGwMbUW46kWlrWwJcE4zCM10Gx2qkFQFf7rabyG6oBr38VWWQaUH09Og9Ng3/iwWMTL1TY7vQSwrR6POkQmj1mRrMUJT//kMf/ukrx+jlv2Ndk0JFXyje5Y2K/1s2xjcTdwA5oHibOekShcHeFv3qyQV9ExrSILK9inyouyfyjUjzcjHEg9j+PO7pWGzrsrFxQpZyCrXso3+uqMld8TyjO9eoRSpgSpD7VTgyVZIKsIjklc/iI0amDPdGfUxYKzdijfk0HmLhQEedCGxD91700OHY/NyjkBkwyKWxVDc/N6hoq18hoDUuaPvYx8Lmt10ygvqWydQcUhx80b/Yh5Wko7WMzozUL/9ZJhekEaoJgPF9nYNm9qocKNLlC2pUkJ22klj+aEyYjI6/FkwWQEGTkX1eMnBUfHUJA/bv9GjpMiEMuAUzVsADufr4SOIuXwZ4scXQPIw3q4DRYqT2M0voubaeQ2ZlfCAqo9C7TouVX1564Kwk4+QPxHQbP2F6UfkVatH7j1VMihoLcHzlFuVICHewNTU+wthkgDVLZsz24EXL1PqfiQeW2k1PcOZITmUorTr8lqKn09odQv09jeDuGCG++1DxDuwmeX7U6hS6n1PNliu28T6FiVwRxCyluH4OJljZKJmdjmie/i4QpGWfJ3dajA8V4HcgGigA72CP0CvsQbIKbZtE2m0CYfmhBxE8vkB+BlPvTw9kgrPpL9bkPait3NP6R8yLSMbw3/0fN2M20OTKlgjbzw6UMMr6QyQ3PDBUeSTlOxECyYRbpRb3zBJ6wCz4+/klztwJ4Wq3n+CoIQh1kO90hVNv0sDwiPh+St9/L1iDLKDU2TE50BzOI+mySDf87fTAlVTQmJ9jko6GeY18yWp3elbXKNKqI0uXEydm PWchRw4T jRemkRJlDjSVb1K6VwGsNhDAhBj19S6MCoV0RGoP5hHM37pID0Uw+ucFiQwEPFu7AfiRNVSrBAVYjBZP6wZtf9hKWCLF/yo2EusR4x+Via3nQ6Q6SztjY+1H7N8HoJ+vFF7ztw4QbjbWrmwKa7Wk5H6H0lGYgQHFS0Pt7u6Uo29BpiHH7TaXeBfH7x8conQHB7brmfFvhP+czJcg1rETgAGfAmAVALz6vXTBzuY/w+ent+0wwP++An010gSmxvl+gYqHK4Q295SEQ5ujCUhKvlzRVpM9MnEwUl1Qs8xlaEDcCd72htdCkp1GrhUHDLdxlRyh49sU0mQvlP5KaAqhvxEmn9o+Hg82t3A/BIoySDzTd+dyZzidIe1sjDgL+jM8pzpRkAk/FKnoUUfAZkfrJGOUXkOxRN3arhMTD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000037, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_adjust modifies a VMA and possibly its neighbors. Write-lock them before making the modifications. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/mmap.c b/mm/mmap.c index f6ca4a87f9e2..1e2154137631 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -614,6 +614,12 @@ inline int vma_expand(struct ma_state *mas, struct vm_area_struct *vma, * The following helper function should be used when such adjustments * are necessary. The "insert" vma (if any) is to be inserted * before we drop the necessary locks. + * 'expand' vma is always locked before it's passed to __vma_adjust() + * from vma_merge() because vma should not change from the moment + * can_vma_merge_{before|after} decision is made. + * 'insert' vma is used only by __split_vma() and it's always a brand + * new vma which is not yet added into mm's vma tree, therefore no need + * to lock it. */ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, unsigned long end, pgoff_t pgoff, struct vm_area_struct *insert, @@ -633,6 +639,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, MA_STATE(mas, &mm->mm_mt, 0, 0); struct vm_area_struct *exporter = NULL, *importer = NULL; + vma_write_lock(vma); + if (next) + vma_write_lock(next); + if (next && !insert) { if (end >= next->vm_end) { /* @@ -676,8 +686,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, * If next doesn't have anon_vma, import from vma after * next, if the vma overlaps with it. */ - if (remove_next == 2 && !next->anon_vma) + if (remove_next == 2 && !next->anon_vma) { exporter = next_next; + if (exporter) + vma_write_lock(exporter); + } } else if (end > next->vm_start) { /* @@ -850,6 +863,8 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start, if (remove_next == 2) { remove_next = 1; next = next_next; + if (next) + vma_write_lock(next); goto again; } } From patchwork Mon Jan 9 20:53:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89823C61DB3 for ; Mon, 9 Jan 2023 20:54:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E0F38E0015; Mon, 9 Jan 2023 15:54:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2914F8E0001; Mon, 9 Jan 2023 15:54:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 132238E0015; Mon, 9 Jan 2023 15:54:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 045D08E0001 for ; Mon, 9 Jan 2023 15:54:37 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D12881C64F3 for ; Mon, 9 Jan 2023 20:54:36 +0000 (UTC) X-FDA: 80336464152.11.261C14F Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf11.hostedemail.com (Postfix) with ESMTP id 3FF1540011 for ; Mon, 9 Jan 2023 20:54:35 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=dyW75mch; spf=pass (imf11.hostedemail.com: domain of 3Cn-8YwYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Cn-8YwYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297675; a=rsa-sha256; cv=none; b=5Tm4C17Hii5UH4biN4TNVjRU0PVw/9UuGAOWHZ4lGMT/xNXnJlT64nPC8jF9WRJ9T3DjOx olsZuERidf4PFY/OOjWAgwrKjfHi2BRbk8PcI/eobIv87IS16RVHvZOfozOAvq/SIXzK8p GxVgsJLa3WiVg7aiYLWNJoR45ytYLy4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=dyW75mch; spf=pass (imf11.hostedemail.com: domain of 3Cn-8YwYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3Cn-8YwYKCB8NPM9I6BJJBG9.7JHGDIPS-HHFQ57F.JMB@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297675; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1sz3d8Q6zHAextbGhc5zNdx6/hdXqibZ5+zjxQEPtcY=; b=YA/sdzSZ36exnabVYIWZkHcwshcohM+gXdaqSOistsAZVNsYE+koL4NPjWDJ9hYYaOuvJe PT8P30w2cpCCKEI6+82zgGpP3Ul49X8TyZ4t8Eidtkf/HLkwXmzUGNRBYyJY8YGrxB6rtu w4zb52xHDYThB7p7R9KOD+ykvHMCC3A= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-487b0bf1117so104147417b3.5 for ; Mon, 09 Jan 2023 12:54:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1sz3d8Q6zHAextbGhc5zNdx6/hdXqibZ5+zjxQEPtcY=; b=dyW75mchIm8T3L3V0Aa5O5essWjp4aFupYS1LFKIISIPYObh/dim0X80nsutKOS2m/ 7vd6AcHVv3F7nL+fFOG4vgOpiulnyZtSLux8AIxMn/Q09ghgqU9hpb0yvL9ry+wKM1Wm UvSXVlZK7VJVWgc1kVtRM87LVA0ol6j3197yqrcZsN3y1TKaMHMTXNPpOvk38N+aLEKf U9xTtNcWgL4MRS7DZ4GXXUuetaMZa2ezppHTDMHc58pzPdKSGBN9qWFGFMwk1DYAuR6m mxxVO9djRkO0qH7ZMi64I+oMQN0AQu1sYjF82YaLSmfJI9fTdyDigl7Wn8SgruW4bLXZ WU7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1sz3d8Q6zHAextbGhc5zNdx6/hdXqibZ5+zjxQEPtcY=; b=m2SOABMXsL4FiLF7t5Cs/ipCjNk5g2NO5GnMcDopCYlpBtKT1yVstQxBbnm2SMObc5 3DdFDo2FDH82qW2Sx8e00/AUu8yDua/F42ZRk27EaoyjnfL5mHnf0w9soAiWEbxnZZa5 6wqO61Y04vxep3BppoZsXnu55jNWaCohdzNLi/GIS4JMrAoqC3e96JZF8x+77h59kIc+ yXCCYFFlFaborUQgwRiWvUlpYtAvoWj3RWIyGUo1GN/gSFkrfGvqSBJrTqRRX2LGo9Rt PR6TDKMh13io2juoSEokm0A6YQRw78AtSMS2bY59//SVQImBK/4i1J0zYN+v7ms02GwC kN8w== X-Gm-Message-State: AFqh2kpbNOJE9AXsBFrByyIEwxzGnvCWwiv6i1nWQMKX02EVH0ejmLwP VG1nGgXczth99LyO+AbtovQ67bH+qL8= X-Google-Smtp-Source: AMrXdXvYJ0MKAPgXdwWnp9E6tLdKAmG2zH31TOsESiFv0D3eqJovHEz4lb9T2t60zhYoFlt31PHaEzcWVdQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:830b:0:b0:731:a583:5571 with SMTP id s11-20020a25830b000000b00731a5835571mr8144189ybk.320.1673297674437; Mon, 09 Jan 2023 12:54:34 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:16 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-22-surenb@google.com> Subject: [PATCH 21/41] mm/mmap: write-lock VMAs affected by VMA expansion From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 3FF1540011 X-Rspamd-Server: rspam01 X-Stat-Signature: q4y9fq9aczob7c9qp48i8apsjf8psgfh X-HE-Tag: 1673297675-95882 X-HE-Meta: U2FsdGVkX1+LUGrl+mvsHaUBDn+fEcAvZ+TrAMF3LlT0yRbJXeYGyz0IzJFQBCwRnjPLnMlV3orMaYZsJ4GPw0N0oSt6rYnYOtGFZGq2N4ahheQAr5SEHhKHw8+HlOu9J6jLjmJWHYIaQjPMZP89aoudlO9PmTTj86dV/yj/GGxPOXtFbZWT2RryrLR5D0WVAN3qEXbM9uSK2WVXpLKhwgIXSNlXhkuMa3IA5GTaz87FrtrnhjNVHae58nhxZgMvRyXDtlr7gli5BNLbvTpL27UWobTIv5Kjqr7mw9RumF8rZ1565DfBfh/yAYAoQdlSAN7XCTBIBJb7dLSJt1nJSDgTnhxUTR1IqhBqveS8HlgUemgLXEfKw81Y5vitYoO5fWEFXMr/jBdQQNPqEOk/m48TKSeAK1dkfZzA4JhU3LYYVMJBBtIbiQGWEfYZm1c0U8Rg6OFGpzCv8CA6Ci2bkCyT0E/trqsOQMynCLkKSZSHIkiTNsNrEUpfvUifGWoHr40DKu985ottQT3ZBSuRfiM1rjLYPYvOSnx5j+P8ldYRl7A4Cc8XEgM4mLU2BqOEj4ZCYExzmjeK/DguYBOYSD/RzvXNGJSVmtzuhKWvp0Bv1nOlF1/FwC8OjUCD4RTrjnVW8r4HSFwGQ327k8onLGc340aaehjNRqj32hXGGKC8KXDAnby1QT39ROd/QvSLBZeX7GffnddnhnenJU6gCkB2Gg/j350a0NsZLzr/d+JOJbWGSqfMFhW5gEJNgXZhQ5UR/Bc9jg0WXpmQm0wpeh0qnCP3flVpbWqHowtckUhM3ga0UJ4VmaPnG0PqeeZahcgYOM/ckAemd0e57pEbH9MzW4xAdN09T3yiToFCNE6sm5Mv79u9S5BX5STw1L7yznFTxkHH1W3p4i8pDUTujFX7N2UcVpka6OVpAHz1DqJm4OFB0cTbralr6WA/g82FEq81+8QTOvtaL5E2SW9 Oi0JlmWJ sE1JJU54ruc0cjozyxcbw2xCoOaC17pQJrszLf1ielIbLT0ye0UdxJS1vmd5SYZsmoQlqT7Z6wUCrLzuQ++DiQWliWB2ui//fo0WZQ4mg+ra0kylJqLuE6FgZGAK0u1PZjhSeSP1VuO7794w49vkovdLo93i9WPQYrLgS1jZRbAjlak2OajZ47OMLiDQLCGNKTj5Jk1E7KYa5XpKtMazha7U8It9HiCW/ZTIogx18Y7NhG5IZ8rzYctZzZWVlaBbWT87V5Lpsf9mHT4XbLaW2Ok95u2lmZbU/W7dVFm7ciLdhqX+oDNAYe4ZPXrtRP02SA4m0DnvVPNA0HA3B+BLdWXjxm8eIszWEpiWNqC4WTgaI0rw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001636, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma_expand changes VMA boundaries and might result in freeing an adjacent VMA. Write-lock affected VMAs to prevent concurrent page faults. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 1e2154137631..ff02cb51e7e7 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -544,6 +544,7 @@ inline int vma_expand(struct ma_state *mas, struct vm_area_struct *vma, if (mas_preallocate(mas, vma, GFP_KERNEL)) goto nomem; + vma_write_lock(vma); vma_adjust_trans_huge(vma, start, end, 0); if (file) { @@ -590,6 +591,7 @@ inline int vma_expand(struct ma_state *mas, struct vm_area_struct *vma, } if (remove_next) { + vma_write_lock(next); if (file) { uprobe_munmap(next, next->vm_start, next->vm_end); fput(file); From patchwork Mon Jan 9 20:53:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3D4FC54EBD for ; Mon, 9 Jan 2023 20:54:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7EA488E0002; Mon, 9 Jan 2023 15:54:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 79AE98E0001; Mon, 9 Jan 2023 15:54:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63B4C8E0002; Mon, 9 Jan 2023 15:54:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5484D8E0001 for ; Mon, 9 Jan 2023 15:54:39 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EEAD1160F32 for ; Mon, 9 Jan 2023 20:54:38 +0000 (UTC) X-FDA: 80336464236.30.977507D Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf17.hostedemail.com (Postfix) with ESMTP id 6781B4000E for ; Mon, 9 Jan 2023 20:54:37 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=AsrsBvZP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3DH-8YwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3DH-8YwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297677; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1FpUUiRmr7dBuhWWRB46HjSP7TgY2quKdzRuNf/hqHM=; b=32NBVGgM0vrrMCLU90IsoWj11lFRvJb3YzGH7AtsMScSigw4pFcb2HQjN314tSRkbdSgYQ +iofrPjIVK5TDo4LWkt1DO0aW47p+UKB+PBw8R43x46T44b3oF87cwItyvU45z5TvpCQew C9tRwk91qzwka6BV7TpfRlQf4mez6jU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=AsrsBvZP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3DH-8YwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3DH-8YwYKCCEPROBK8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297677; a=rsa-sha256; cv=none; b=Di/PdYIpAOLGW1QnZRkE5UWALl2tG6lCQz4DIx8y+rEtau91AB4voq8wcHof51eB/m45WG sZA7DkQEGnFNkmqNb3tSZtyavXRVQw1H65ywY2y5Q7wDvwC+uN/P25cH6vhNrGv43Pv/HM Xbx0JTj880AF7iM+r9bxKti//hFL/Xk= Received: by mail-yb1-f201.google.com with SMTP id v4-20020a056902028400b007b075879242so10390780ybh.22 for ; Mon, 09 Jan 2023 12:54:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1FpUUiRmr7dBuhWWRB46HjSP7TgY2quKdzRuNf/hqHM=; b=AsrsBvZP9aQHF3L5Nl/u1fB+D2bm7e0YjzSCvgbBaPzVvDqUAEV/bKN3f/FWWFnJld y54pIfCVv+Sf28j1GT6GnDmjTV3QxG0dtIi870ieojG97R1YrLaN1+Hnnh1RqNC6IKhV FDsy8k4m0XBTPaqC3JVLB2amGI721PZ/e3uTJFIVXTdjA/KJiKwGIaizGxyvcp5J8QaW 5xEAO0mZOLLdDxM+ya/sdbkzi1vVwSyWReOOtmKkNqv4WGZ0uHxmN44Y8NrueULgHAw2 BBJ3cZtaXuduG2TgX3HtPnChX2RwWmECLeYtrImIURhb/vK80eEmATn6FHpKHV2WHJD9 N1QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1FpUUiRmr7dBuhWWRB46HjSP7TgY2quKdzRuNf/hqHM=; b=tCkRRqTNJBJx71USfluXtNxtFwKyW+bPiPKaR5LrX9Wv9lzfLkCGDxm1hPwVYEqtwU PH1GwV3u8vsxp+yekBWnEqUtcPVV1azAt++cEyRP6ueyHSLf/frnvkaeLgu7Vr99HQZJ mIHA0OFMowVAxXmgpd3Ioy4s81e02BMjMET30ww3lAq0P4+fh+o+oBCxGTyDwLu4Iyvs eLFQVgkgD+WBgu3MGOm4svn6HlB1DY9pEWb4OhSH0CDhV25ZH+eoSvdcAQNb2iANzJmg hRHWwU5o7knm8O7TuG9HAtqHmfcDRd2MFqXRBX/O5kMaT+FXdTlewZoI7HUxqw5FP5Mu jr1Q== X-Gm-Message-State: AFqh2krP4Z2EC7j90glCHmoG2mw1AeVRvY5VVceVS2PiPj+ufJT6X2i9 RvyIcXP9yJxdnwqcyYjWP57q2DQr7Kw= X-Google-Smtp-Source: AMrXdXtAgZu+pAoAaYuQQAsM5E55paYeoDQZrstt2b6LAIgKd1Hk/SubADrA5GRLvEWllXjlmSi56GzDLGU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:4646:0:b0:799:3955:201f with SMTP id t67-20020a254646000000b007993955201fmr3367922yba.94.1673297676537; Mon, 09 Jan 2023 12:54:36 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:17 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-23-surenb@google.com> Subject: [PATCH 22/41] mm/mremap: write-lock VMA while remapping it to a new address range From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 6781B4000E X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: xku5rcoh37oabrkorut683sugmnrxufn X-HE-Tag: 1673297677-445205 X-HE-Meta: U2FsdGVkX1/ZlbCJxlnKjx1j4Tyn87+D1j1xp0mZXg2AdEK7Z+qBIteS3OuJ2p3c8qcdrlZORSTKnDX+t0ZpHgX/omfVMR8tepESaEKgV5PiRbJjboiYUT6FpXeht719hc3bxG88Asi4RTVd6Jqby4DcQSnQeV6T4+NDc29CwO+J5sFmYFaQbyXhzg1Ds/kgbzUHpxyocWrcnrE4ZkO+MKHodqOL/6GwrWMvfuiaCoEsYU6CXdBBsTBNbKt/iula9Ek/0uERhh8KV24zQfXNpQsuglF/Nihy2l8Iy3D14U03ismWSgIEl7LGH1ekKCEkgtkexcDYdlkHVEXy9xqkssFODXxpjZBocHxW9TDTWdKbjkauWIGXJeE3+W/gw/bG/W6PGfjKWAm3tEJqhn0fNwjw/uvllviP9bfpiG/kdSTTYziWUhQ8AbdLzuBeegqOI+0qDa4ATNj15yNgfGXbyG6vYiOqNXTbRp60xMKd4mxqQtO3lu5ItkyWzTMBCoGTY/xPUkY9Hrpk/myK6F5nLDD77otsymGYl3Xs54bvkrVCMVS25Z6RSz45Hd6h0gQLDyqYX4+ex3S/43x/7+qxabWv2/dtJg4R7I0JYM/QcQbo8bKOQ7GKAxvAV2fqYaWdRBo71YCRbp1gkOOrIBwFKV7hamop3+k46YiRm5jH1pUoQeuBATpKE4Hl0z1xtthbgPJiot7RVRQj3VQdBxDrhCrD3PL153BhFyoIUJUeGI7nrPDzSyEq3L3S1mmk5ImCx/xOkXwo8c6WjL/rS18YP71Dp0AizXOlRdTCJmrEl1ZnrD/MEMohXcGCoHXVKrWI+AbgrMYx2KhAD7Z7ZlEmPkrBfVovlPPN1YrvS+242pACc2OfezLbM25dHMW1jhGomvk8QlFeR7TO3O1nPliWqy6noM1ET13/UbFv4zwok8lmAoHrPxBp3Gy3fghcxf9CjSafiY+8SVqJpQ/ifms FQPbtMru 2edHF2ewykzGsSTWawosJrrU/O+H52YPSBMq83QD+cbAAyUcq1tgy9tVqmyASfbdvoAmynhz8NBd/dbHCTqK8QXnQFVfAKOe3JUMAhK7mBVe/Lk0ugausVVuxWqlLD9UIxYlOBy2gj9cEH/4rUtjuKm6kdaFtriMMlkzW6ZZEQGY4zEuMlTP39rMTIb9k5Q3V4jcE2pNK22oyx7sW2CgTIzHRMXIdxTY1F8/g49R4SS6c5Xs0f5lyEqf4XIzdsh8vfNUQnvzk0V3rO6UwFYccCHGw9TzyWHHDW5q4xHrdEn1wsGKEb78OxQewqaKv1jeLVUGl5J6lRvcsfW/kpyP84BEvIDovBDhcR6nez4oeC21TpFE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Write-lock VMA as locked before copying it and when copy_vma produces a new VMA. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/mmap.c | 1 + mm/mremap.c | 1 + 2 files changed, 2 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index ff02cb51e7e7..da1908730828 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3261,6 +3261,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, get_file(new_vma->vm_file); if (new_vma->vm_ops && new_vma->vm_ops->open) new_vma->vm_ops->open(new_vma); + vma_write_lock(new_vma); if (vma_link(mm, new_vma)) goto out_vma_link; *need_rmap_locks = false; diff --git a/mm/mremap.c b/mm/mremap.c index 2ccdd1561f5b..d24a79bcb1a1 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -622,6 +622,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; } + vma_write_lock(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); From patchwork Mon Jan 9 20:53:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BE23C61DB3 for ; Mon, 9 Jan 2023 20:54:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9BF118E0003; Mon, 9 Jan 2023 15:54:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 96EFF8E0001; Mon, 9 Jan 2023 15:54:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80FED8E0003; Mon, 9 Jan 2023 15:54:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 70F708E0001 for ; Mon, 9 Jan 2023 15:54:42 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 466701C65D7 for ; Mon, 9 Jan 2023 20:54:42 +0000 (UTC) X-FDA: 80336464404.05.B147BC0 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf24.hostedemail.com (Postfix) with ESMTP id ADC89180011 for ; Mon, 9 Jan 2023 20:54:40 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pKSCW4f0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3D3-8YwYKCCQSURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3D3-8YwYKCCQSURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297680; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=olOpibg5eXy7KG3cFGJtGZQRi2wnSYuvllMlsZALFjE=; b=w89lltjnCijcX0rvuy/3bD3OEC648VaWO/js8hFlIkEI4HFqgDdfX5ny/tdi0dT72O7DFd anZ7GdddJ9IVr9cNZ7g5qyH1f+azeZTGeoyJQKnOSA/kjSBq7xpSaBIJJEsxiS2ZQ5DUHw sbcUIJxnb0FN7QdzyCTDHem+spETaMs= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=pKSCW4f0; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3D3-8YwYKCCQSURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3D3-8YwYKCCQSURENBGOOGLE.COMLINUX-MMKVACK.ORG@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297680; a=rsa-sha256; cv=none; b=aU8VeMyqTDzC0aH3btq/eaTNQ6ZaZipjpjYMod7LgG+PzuohGvM3Wht4mh95yK5+1CFxI+ bSX++s8IVW3Dhx5aiOUrhUsg6tKzGQpCuh3zGwez9j6kI5PpySFBLOJ5dBPgow8gyDVZq3 SxfHAWWsEUbsr6kKcONLA+6aE5Ts7yM= Received: by mail-pl1-f202.google.com with SMTP id l17-20020a170902f69100b001928d6b3efcso6963557plg.2 for ; Mon, 09 Jan 2023 12:54:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=olOpibg5eXy7KG3cFGJtGZQRi2wnSYuvllMlsZALFjE=; b=pKSCW4f0d77MorxMhTYeMa3Bkvztzi2l/11p+8AD9JlAcd7O/4nLaK3tMdO/JZYFZO oqL+3k7pxb64mumhhSbnD+8oqAK9T+YZbtnVKYdyWwz1xTl0lxuCTGJlmjyD4W6tkdoJ ScE8xkE+709mZSvnfMafhqWZz0JUfC8PX8Gqgq4hVWAiFqsbly88nJnbf2OtvsEjRJ+V Bh1jSeLHcoArd7dhSosnJ19E8FECzKPk1IhOzNYJUOL7f9tKHCqrIEP0OXg1wGsfV/sX JopWIX7MNjYmVxKWpBhB8gnFt1KMXJN0MSlcK65fQjg6FKg5IorU4J4ePrdeGZ792swl N3JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=olOpibg5eXy7KG3cFGJtGZQRi2wnSYuvllMlsZALFjE=; b=NjJaYaBOYnW2FcbRzqHdSWU9Yeo9vpnwL5lHZL6awyqabwsHskqo8huDfybmkuwiFY FQHJrTE0aMP0ph8VNbHaWgc8aKcCSZJH8u/c3ZNAFhMNv7/pWOl8bcbyjnXtIx31KEkw G9zfhlgKNEYEsRWDDdTwXmdGZwA1FUpRFKgek8jQmcHsnW2oh7kdtbB+l/9FgMn0fcCm 4r0/NGWF+LxwVFRD+QmaKYZE3YfsdHSh1b2XNFiWaQkbS6/+hEkPa1uXa3BP2NGamIDr sJDAqV2jJG/uvV4o6u6rm/rZAmyrIUhqSVgot48MIwapEMOPkEGP6aJyegR7KboUGXqL zE+w== X-Gm-Message-State: AFqh2korTnx7kF2p5+PnImk1EkwVL6fjWSnPKLzESnK/XJ6M89NIaCGY jpBASjooTHZ2TLYQgjl2c9ngV5LeUjo= X-Google-Smtp-Source: AMrXdXsFsv+WY6uB2QAxQKVey5zkUH2AXIcpYb3YNAv+Gej95uP4vEcIzk/UCgPyK193sduP1xpoK0p/ZF8= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a62:1501:0:b0:589:8362:c7ce with SMTP id 1-20020a621501000000b005898362c7cemr251559pfv.21.1673297679447; Mon, 09 Jan 2023 12:54:39 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:18 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-24-surenb@google.com> Subject: [PATCH 23/41] mm: write-lock VMAs before removing them from VMA tree From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ADC89180011 X-Stat-Signature: 7sk5hhpnj5o6mxm3ifq6aubzugrnbjmt X-HE-Tag: 1673297680-394965 X-HE-Meta: U2FsdGVkX18cL28ycL0/SU3Gl06Vg8gYdVFKdJ+anwAN4jxNlUk8+g4WQkOTbZOAV7ZmPfT+G9jp1jcaERR8JR2aGHZL6SsADkkYJ1W1/Dm3mLa0zBqkoK/La9FXpzmGnquTXhuVs8VMSGCet15U/Hxc6Xr4leXrSqhiKOVxuFkLorGuuhbMPc5uURFihNuK2RcktKJIToopiKt5mP0gf/EwUEU8JAEjkO1cqCI2BoAtXc36F6ya6MKFRVysCbyV+iPaR5dE+EQFB0VMtQFmt6kci7nBoxU9tD25YtfWhCdw4V7ta8Sqie70vtz08dC2gIhDdMO5U5XSm8KKbsq91IVMIw6hXEOZP/oF4tmqTrIjdVrr3oG1im5/0skCGMRfIafM3siPEF7QRLykRl/UT2cnbvNyAC6gFZve3mltr2/EeCg5bSWLANwEePqhfd3Kg1u9kdxvjrXvzUz8Is3BtndnoGIx0c6LPRfl6pFIl4gJrmJkpnF7dyCkA7KKmiOafySRi3Z4960zfoAGPiYhgEPKe7LEzlpjIIKGSg+Y9u3njvoX0xYIuGLD0RoMSNZPLFIhshcKR5lcCE+qBSFkWh7sragBVnweX8LaPW3CXNGZyJkjo+rBRMweKX6r7G8QFokQSXzYy0S6PL+PsRC1kstKGf4Bt5P/UV0CJrHubFfL1vH6pYYmFuv/IjH4zctCQ2QpVE5Ko0xButy+LekPVDbC8DbBWwuW76SwZ8k4uv+PUwVOaMQf833LLUbXvH3XHExYNMpvCBjf6L5q7o28epKoi/1XJ+XPadTIHonDH9a9ZraiTek6f60oq7SaTIB0DyJOj/NABqe8xpFNuS+AenS6IN3TmZkJwY3PIt0hXG4kSkZJu8MykSqF0V8J494r4xVXgZxqYcdUHjPRddDw2xkCqSqACk/guEA5RcG48gvpTOq8RjuR7UpU5Q59BQQD9Fa+m5bfmTuWtb+J6YA l67kZv3q abu5S4wp/JJVDsRpBYaprGtfHiyo20BHY55j0xF0CWL0qAfs3S5rDxDWm3JAo8nYRuY2TeNmXGRgEXZJrj9mSnR3jiCz/sBIZp3uPFVWOQVU8FNxPMC6o4ZnqK+Aw4cy6e/otuX7fHWQZYydJO67jJNPrgTPbLvQhPjmBTN/j/50ZYRssM08djT3xbVw9nU5GDw7yDC0GTQf49YcV5enYyXqSSmIY1phcX+qiLCGoPoXcUkzyXOvzcHyjZG5evGeA1+FpGRdORqyWd1HPyWWUskNs/p/W0uAgZPPXbCYv2V2gN3DTujg7Lx5l+XU/SkoNwWgfqxgMwgPdil9ywYzWgGIMJBBi5t064KjkNjjDzsNcSPU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001920, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Write-locking VMAs before isolating them ensures that page fault handlers don't operate on isolated VMAs. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 2 ++ mm/nommu.c | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index da1908730828..be289e0b693b 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -448,6 +448,7 @@ void vma_mas_store(struct vm_area_struct *vma, struct ma_state *mas) */ void vma_mas_remove(struct vm_area_struct *vma, struct ma_state *mas) { + vma_write_lock(vma); trace_vma_mas_szero(mas->tree, vma->vm_start, vma->vm_end - 1); mas->index = vma->vm_start; mas->last = vma->vm_end - 1; @@ -2300,6 +2301,7 @@ int split_vma(struct mm_struct *mm, struct vm_area_struct *vma, static inline int munmap_sidetree(struct vm_area_struct *vma, struct ma_state *mas_detach) { + vma_write_lock(vma); mas_set_range(mas_detach, vma->vm_start, vma->vm_end - 1); if (mas_store_gfp(mas_detach, vma, GFP_KERNEL)) return -ENOMEM; diff --git a/mm/nommu.c b/mm/nommu.c index b3154357ced5..7ae91337ef14 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -552,6 +552,7 @@ void vma_mas_store(struct vm_area_struct *vma, struct ma_state *mas) void vma_mas_remove(struct vm_area_struct *vma, struct ma_state *mas) { + vma_write_lock(vma); mas->index = vma->vm_start; mas->last = vma->vm_end - 1; mas_store_prealloc(mas, NULL); @@ -1551,6 +1552,10 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); for_each_vma(vmi, vma) { cleanup_vma_from_mm(vma); + /* + * No need to lock VMA because this is the only mm user and no + * page fault handled can race with it. + */ delete_vma(mm, vma); cond_resched(); } From patchwork Mon Jan 9 20:53:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094290 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3072DC67871 for ; Mon, 9 Jan 2023 20:54:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BBE3E8E0005; Mon, 9 Jan 2023 15:54:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B449D8E0001; Mon, 9 Jan 2023 15:54:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BF528E0005; Mon, 9 Jan 2023 15:54:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 888658E0001 for ; Mon, 9 Jan 2023 15:54:44 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5F632A0BB8 for ; Mon, 9 Jan 2023 20:54:44 +0000 (UTC) X-FDA: 80336464488.22.0AD663C Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id AFA3618001B for ; Mon, 9 Jan 2023 20:54:42 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=PUe2nQQD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3EX-8YwYKCCYUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3EX-8YwYKCCYUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297682; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AwPKUNzvgMd9Kf7LcbfBV8kCxW3mIJvnnhSUrm6UVIM=; b=g4eXrXNw3rjlVHHKV02UmqyE5h9/zJpi9O6bqsBKARBiny8VmVUpRMBu9Lc9mJSIdyPxqe y7o6nMK1wySJY68c1yJNAuJF/XM8wKV6ZDzTfDYyyu6e6qZ+yB6VZ580DpkXSi58DtTBhp NrjTZsBdWh1alTjoUYQBkXN8GvzPC5g= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=PUe2nQQD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3EX-8YwYKCCYUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3EX-8YwYKCCYUWTGPDIQQING.EQONKPWZ-OOMXCEM.QTI@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297682; a=rsa-sha256; cv=none; b=s79/LtbdjRHFrXK8Hp7Eem27njsx14/B1zhOLbt/gZfRdjUYLJzU7SNxsiUL3mFQuscxND 1qYDMaF0BM/+y7gxo8y+gAW7BEGlPDbN7s0CZS3exHJNzaFTGAW2zJJTCiamazD11NVPPl MN5pHJKDMsREEksHZw10fYH8BCPhJiI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-46839d9ca5dso104905777b3.16 for ; Mon, 09 Jan 2023 12:54:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AwPKUNzvgMd9Kf7LcbfBV8kCxW3mIJvnnhSUrm6UVIM=; b=PUe2nQQDxcDIZZYDvSnvU7x/WgxDknYnlEY7L4Z8C3iDtsfKJ6ipNpTRC5N0+o397w Zlr5LNQCm8jIObT2qwQm+iFFbmfCf3so91ezlzHPCZn3+26NE3G0VG8VYTX74hVikOaP hXnC1hgxqMkKtkdKCm4ja7+1OhhfuKZ8QD3Yt7TKznRnTkaA7k7w1KBOtddu/hOGuQp+ p5yPJ6DKvl3ugNtnIXuytmcOf3bXiekUrutBQkFLl092Px87ewbl/fVBINgSpD18bwdH tuw54xfQPY3gshaLVXYSV9JTTyWxRw4/COwHM+KjA25VHZQUSDq3VtesXUsv3nuWcWdU OueA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AwPKUNzvgMd9Kf7LcbfBV8kCxW3mIJvnnhSUrm6UVIM=; b=NnHs+WL/Bmh2WGaRAPaq2w8wnuByuptZ1sb4qtDSyDApe8IXO6nA0Iy50SRtSnsBz+ gzw3LAe6R1Zsfyc3i/KrRAwvv3wetMBVTiXkMWfrEnPnAy0nBUDp/dIRdXU/UK0AZo82 RTiqST+eSDm/MRSWKQKw6eDDi1y1DhqOx9Gu5EEXKO+8utm4HXi2M04b/8IUeFqv5X1/ FITjg4HmzJJvtLmQ7IWX4hBA5LBTH6wH3OpnW7E0W1V0lcBmsuP3e16i9ob5xW1GYa56 E0S9M7CYphfNDEIGc2pTDU9PE+6ea+pKU6c/XrPgBTIGl0PNNAwHLain42GYHgS5cMGW wkow== X-Gm-Message-State: AFqh2koYxJvCfhiM1yjiS6hoV6ixDd/6SFiz91DjZ28mK8KqOuuf6Ipv kevDEk0CV87D1RiDs4ODEHMKXCejThQ= X-Google-Smtp-Source: AMrXdXsf9qNWBP/q4HbIX0JjhZtra/GO9q6jv0FDi4j+zY41mh+XeX3qFvEfYkTJ9YCl7ccm1I7VEz8cc3E= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:4702:0:b0:4cd:f764:1911 with SMTP id u2-20020a814702000000b004cdf7641911mr673145ywa.403.1673297681920; Mon, 09 Jan 2023 12:54:41 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:19 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-25-surenb@google.com> Subject: [PATCH 24/41] mm: conditionally write-lock VMA in free_pgtables From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: AFA3618001B X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: e6dhc8df37e757jb3ywc1uwsbcdk9fs9 X-HE-Tag: 1673297682-12858 X-HE-Meta: U2FsdGVkX1//wppgRBZioUa3h7cVEPpnjgcBlgqdXzd5ePgMeNfJIFR5vLrHtBbnMdeFP3hPHXlSQx0UIAvCgngupDlQcbhWw/08tyR+IOaoWU4wPNh/PXCsWotrWH021ZOTRZDTqy/LlYgGNSwpIAbwtgTvaZ38wcSKTDKWewp1TO5otunjm0Dq26pwUawx1Q3Htrb3tpbOZ8iC7Rujply927paSMYY/8WDRQKS77iVhfn4ZYRnZh5XF2vWuk6SeMvJMkkyOe59UEviLwMT617h+A6nfcydbST9SYWYmV2nvdrLOWp5biMCBruh3hFRWIQtLzH5PWpUkdOZO/qyW44qjEgviBXmAtHZ8D4kU0caOR7kSzlxRlYiubvYGfHttgspdh3a1F1H1fge+NInEhNmvxxfnUtcIElfiT6b41OTtEPWy67C/VoyH3xlHIukkULvctxWRC/eiiMFMmPy4Z0qTH6NGa7+cwTxcXBpPwBaG8U1aN8WY6LzKnh625vhsBbLtAVu0O5oGfJKJfDxLBfwYlgLzb0N0PYYnPUgL8TXkRxkIHs3Pf8Q1hV+6qlGviKax/7ZST2h2ptPr2xrDFxwvCbU7lm6l9IghB1GoiPLziGdf6dLpB05XLbELm7IFdOd1mGtB8T4mPuha/D5fu1mWfCvkX1mFUOqCCoBO0rkUg+zQMSuIs4k5GOa+qpJxpaQtokdhQ9f5JWlPMxgwko1AeW4sLaEHk6KKV31IhLdeyE5b5miHg1wlca8ED2YltPxGtJGALEKbef3MxCLhpWeJDtXDctXNzSPwOqV1CEekiHB4VTE9u/qBk2iHzmKAzM+DGUGXpn0Ur0tkfKMV8f/qqYiiPNOoGIR3KKYqGkiO5zA0LZp1SgqgFyouM83GUYMlkDlov92h8MSaLeu0dTwpfqx58gwnmA3BbUfoPmoT6wnx1H7zbU9QM4mBFIkqWj5A2DjrHeN5dRoe+f b4qi/L6g WzFk2T8anzZy4kBigEC89FNW+wi/zQC4x845iynXmxeoPd+ISyCVvCJ5eBNBny6NDtWycWCMfLMHvEyP55tDQMm78DkJPzSlywF312jbx9ElfwXDgbcP8da7nD4tz2/HHBiGOqjuaOnZ5wH8LZQvIdGvHdesha6Z4h65BcuiaeYia46M6XfRcpBPqRLLgM6xHlmwdiKjpjquvAdgwwa2y6FQMaZccSP37M5WGg543VisklWgC5vINB38zske/zVBdz9uvr692ufeu3lTZbNL0ORiIBlO/u1pV65vtO2jDmsQ4xmb6weC3m8ea0tgx9gdpCAA23F2zYRsc3EyuPTZOvpGfWq1UkQ3lZcrk5JFRJtLql5558w8S+WzUZ7v2/VqFHUBhNWr2XDXfTxxymogkVJm0H7H2q8bEqun+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Normally free_pgtables needs to lock affected VMAs except for the case when VMAs were isolated under VMA write-lock. munmap() does just that, isolating while holding appropriate locks and then downgrading mmap_lock and dropping per-VMA locks before freeing page tables. Add a parameter to free_pgtables and unmap_region for such scenario. Signed-off-by: Suren Baghdasaryan --- mm/internal.h | 2 +- mm/memory.c | 6 +++++- mm/mmap.c | 18 ++++++++++++------ 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index bcf75a8b032d..5ea4ff1a70e7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -87,7 +87,7 @@ void folio_activate(struct folio *folio); void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long floor, - unsigned long ceiling); + unsigned long ceiling, bool lock_vma); void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte); struct zap_details; diff --git a/mm/memory.c b/mm/memory.c index 2fabf89b2be9..9ece18548db1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -348,7 +348,7 @@ void free_pgd_range(struct mmu_gather *tlb, void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long floor, - unsigned long ceiling) + unsigned long ceiling, bool lock_vma) { MA_STATE(mas, mt, vma->vm_end, vma->vm_end); @@ -366,6 +366,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, * Hide vma from rmap and truncate_pagecache before freeing * pgtables */ + if (lock_vma) + vma_write_lock(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); @@ -380,6 +382,8 @@ void free_pgtables(struct mmu_gather *tlb, struct maple_tree *mt, && !is_vm_hugetlb_page(next)) { vma = next; next = mas_find(&mas, ceiling - 1); + if (lock_vma) + vma_write_lock(vma); unlink_anon_vmas(vma); unlink_file_vma(vma); } diff --git a/mm/mmap.c b/mm/mmap.c index be289e0b693b..0d767ce043af 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -78,7 +78,7 @@ core_param(ignore_rlimit_data, ignore_rlimit_data, bool, 0644); static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, struct vm_area_struct *vma, struct vm_area_struct *prev, struct vm_area_struct *next, unsigned long start, - unsigned long end); + unsigned long end, bool lock_vma); static pgprot_t vm_pgprot_modify(pgprot_t oldprot, unsigned long vm_flags) { @@ -2202,7 +2202,7 @@ static inline void remove_mt(struct mm_struct *mm, struct ma_state *mas) static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, struct vm_area_struct *vma, struct vm_area_struct *prev, struct vm_area_struct *next, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, bool lock_vma) { struct mmu_gather tlb; @@ -2211,7 +2211,8 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, update_hiwater_rss(mm); unmap_vmas(&tlb, mt, vma, start, end); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, - next ? next->vm_start : USER_PGTABLES_CEILING); + next ? next->vm_start : USER_PGTABLES_CEILING, + lock_vma); tlb_finish_mmu(&tlb); } @@ -2468,7 +2469,11 @@ do_mas_align_munmap(struct ma_state *mas, struct vm_area_struct *vma, mmap_write_downgrade(mm); } - unmap_region(mm, &mt_detach, vma, prev, next, start, end); + /* + * We can free page tables without locking the vmas because they were + * isolated before we downgraded mmap_lock and dropped per-vma locks. + */ + unmap_region(mm, &mt_detach, vma, prev, next, start, end, !downgrade); /* Statistics and freeing VMAs */ mas_set(&mas_detach, start); remove_mt(mm, &mas_detach); @@ -2785,7 +2790,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma->vm_file = NULL; /* Undo any partial mapping done by a device driver. */ - unmap_region(mm, mas.tree, vma, prev, next, vma->vm_start, vma->vm_end); + unmap_region(mm, mas.tree, vma, prev, next, vma->vm_start, vma->vm_end, + true); if (file && (vm_flags & VM_SHARED)) mapping_unmap_writable(file->f_mapping); free_vma: @@ -3130,7 +3136,7 @@ void exit_mmap(struct mm_struct *mm) mmap_write_lock(mm); mt_clear_in_rcu(&mm->mm_mt); free_pgtables(&tlb, &mm->mm_mt, vma, FIRST_USER_ADDRESS, - USER_PGTABLES_CEILING); + USER_PGTABLES_CEILING, true); tlb_finish_mmu(&tlb); /* From patchwork Mon Jan 9 20:53:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094291 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65852C61DB3 for ; Mon, 9 Jan 2023 20:54:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 04BB28E0014; Mon, 9 Jan 2023 15:54:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F3CDA8E0001; Mon, 9 Jan 2023 15:54:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DDE2C8E0014; Mon, 9 Jan 2023 15:54:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CF8F48E0001 for ; Mon, 9 Jan 2023 15:54:46 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A7501140C38 for ; Mon, 9 Jan 2023 20:54:46 +0000 (UTC) X-FDA: 80336464572.09.62AD99D Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 15898180005 for ; Mon, 9 Jan 2023 20:54:44 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="nfh/0wSN"; spf=pass (imf16.hostedemail.com: domain of 3FH-8YwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3FH-8YwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297685; a=rsa-sha256; cv=none; b=GR2jWStuNhRdfriTvpm+qujse0AmM1HHNmvepntT/EeHrK8DQ5RHCpTkE7tfA2GroWXIuy XZw7jU9KqtMCCmcDwfvTAtg7RcQqTO/HH6TMj5LFpGy2GX9IcLgD0nLP89Cykrbm6ZQ8Ym O+wWjQf2YhRPDGdTiKWAxmbJlCt3lGk= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="nfh/0wSN"; spf=pass (imf16.hostedemail.com: domain of 3FH-8YwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3FH-8YwYKCCkXZWJSGLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=89ITsq4nUiN8d4MMzUNaP/VKvoRvGlIDiLzssAyWMSw=; b=kbw/Y55RQpVoQrUTAL7I5z+2Nckmw4VJRaH4GC3J8uO7JoP3iHSZy1CJ0Y92IC266EHk1p FDEpTHNGbP8Ep2D5LjScgWa63pm1OK/ZnayNoRhQLTpPqfFWtaWVjXvNFySoNzVDQcsscu uHKHaoIIVawX79V0esEMPJIdhryWYhU= Received: by mail-yb1-f202.google.com with SMTP id w9-20020a05690210c900b007b20e8d0c99so10270520ybu.0 for ; Mon, 09 Jan 2023 12:54:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=89ITsq4nUiN8d4MMzUNaP/VKvoRvGlIDiLzssAyWMSw=; b=nfh/0wSN7du3H+jbnftiNBYsH0L/ZzvAjrQDNKD6/Ux50j22Vuiz0S8HvVp3TvO1vo K/T56ndCuYOiwpC071dddEQYgomwd4Y1mEzWyHgdRsI4Zm5Szg4pGbhhjtZTIjjKwBZ2 DuLjlMuvIIQ0lWiW9Nfxk/1vqmTgUUR+Vpz2r/N+XxIUJS4L6MBvWuo91YhKUizAdsFD /Z7UYgJwWP6aH5EHzLl9AB9L/N1SliUIF0AbaZQSGUEgi4Q4Dc2tmxMGPV7cLiqwxdNW 0MiPjg6rJc2LN6ARMi2zMWyKxfOe4Y81VTcKlWoejCss5A49BpopZcqw0LB7HbZRjJwk DoGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=89ITsq4nUiN8d4MMzUNaP/VKvoRvGlIDiLzssAyWMSw=; b=5r4aOVpuh2km018nNiz6b0BNfQUQKQuNeNJNf7blY9IIDD6ce/9UxNfflE6hFpigYk DkGHB4bikLvM95O3AMYglZL08pjnt9GEWAtQxMzBVyWMrIg8jKaOEiSvN/2a+lNYCUKr Nq6AGXCbXuzzaQRXncTLPZykrYLyzMMu+N0Xn6LTGrz4KUtyeaj6xXpPskf+6luHs/Zn G7rcd2oYyIaASEOWRa9xYLi3hb5rmlvX6rTeS/MXCavBkiZXU4ysPnarG4KWrEmNzU03 SArb1ulwVQUjO3LEeE1nTke9R+tz2baHtK0wnRBUwiWwyPFlZCSGfW4Wo3b5R/ulPXT/ m16w== X-Gm-Message-State: AFqh2kqQJ++PWgXOajfBx7705kqEFyMVcxiUg5TXailkYWyTib6nppu/ 7zeYDWPk6dRWwrCia1vJtaJUytJQOXI= X-Google-Smtp-Source: AMrXdXvWplK/QqycNgSSdeBDfganGdyEUrMbsmwVLuA4csVljoZeQRUCGyXX2t9p/6nnKr16T46cKc1FR2o= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:eacb:0:b0:485:126c:3834 with SMTP id t194-20020a0deacb000000b00485126c3834mr6100451ywe.360.1673297684211; Mon, 09 Jan 2023 12:54:44 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:20 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-26-surenb@google.com> Subject: [PATCH 25/41] mm/mmap: write-lock adjacent VMAs if they can grow into unmapped area From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 15898180005 X-Stat-Signature: fee5ck33bds4rcgytap6hpybatbxjfnm X-HE-Tag: 1673297684-219156 X-HE-Meta: U2FsdGVkX19BwdB6uSBBDYSRh0c8ThIlIOZ3fFK8e1mXc0tB5fPaiMwG+lzBAmERinN5PbhnKWKQr5qsuYJCIThqlY7DgiKkt9I8okFan0lGX9NmXmVuhsihp+26VMTjN2Sw7t6B83ZdCHZBn+o7msHZqBRb5Leiv21D8vzMFh6uRXeNk8TK3LY1I4vwE5V18KyTyQxTH8mOjYo76Xx20+WzO5z41Dm8jpafF6ylHKWYc16plN0dxBYArnS6uc1ilLQ6hnAVq7GwOT14y8fK8fTKDobc+TDFFW4ahjQRmlZvnWYN6c93OOcewOD+tiufB9BckP+lLYzDBHLQuKXEpe8l0+bNV3m4gCL19TSUl8ILz6anALBpwMSQuC29fPCjIXUKutFLvQfuoTYQN+mqzK1soR2KXrf68093GAIs9pmZ0+BFU4u064/3JqAYnUx8PO0kcFEO/he8kl6LWGVwcMRodE0+09sq9njDR+VMQA5L0ywRx2hIP8AnkDTjfsNinwxhjRaur3RYg/aubob6+VVMfVvEseIgjDfWzebnDEKymQ8sdMImTy+SSwZIEnSx6mU+WksEJKl5mAT1ktpZdwSrLPJ9pCVrx/5AmIHgGJG4eNsTnRESu7jYf7kW4Yzz6XbjqbG7V7NjHNqI7KdkuLsWxV9CFdfOJYhlp6687NeRkbEyvSn0FVuWts8eD5WaQ1aB6Ab9HMCoite4h3D6+cuK0HuWMUgcnNuDxnLDPLPAjRFXaXZfrWRlZ1wJBlsEnqooLN2dmBP0RlITs5cj8Z47H9mVKLcjU4jnhRJoZLjVxZYM/At9N21R7Fnrp/Z/2Snj5b8uMW56tCIchu3QZ0CP7yWSOLObq6BWB7wVsHAeN2NbfE3tsj+TgLo/0kLSlRdsApejuDbLgyYoQ8oj523zSV6bSBFeLGlkX4mg9FSd+YTKUjPEmxNxFkl3DtCvwpdVXiIfA2qy8naH5mD qAqUhWm6 znt/eL3KVuH8wJAR7qzeUYl/0BdXlRyf6qsJyXgH2SpuDq8wmKVX4uZLlBxXh0VxegsNVJnDR1CiAjKvLUqL+sUnfli+PhLtM9fqWMMYqep4GoWphZ1g/HTdMBE+30cZAy7X3W8JVelDnlrJ2+6PPBPjze0ukZYu2FK1lYrb2vK8q2V5EjX61PdTdfUHeAfwZNSIxDYQ6FdXszNtNz/vhqdiwy5tJU/qAHTvyhkpQl2m4qwMalURhfwgMuYDXOrFNMG+kmHMyLhcuQc+LtssMLB28rcHliR1icK0hge/zk9oOl9OwXvoOtODykwb/C91khTkwEJib7AcsCs1hDB7T9GRadly8x1UptonLL8hhUdCRQ1c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.001531, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: While unmapping VMAs, adjacent VMAs might be able to grow into the area being unmapped. In such cases write-lock adjacent VMAs to prevent this growth. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/mmap.c b/mm/mmap.c index 0d767ce043af..30c7d1c5206e 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2461,11 +2461,13 @@ do_mas_align_munmap(struct ma_state *mas, struct vm_area_struct *vma, * down_read(mmap_lock) and collide with the VMA we are about to unmap. */ if (downgrade) { - if (next && (next->vm_flags & VM_GROWSDOWN)) + if (next && (next->vm_flags & VM_GROWSDOWN)) { + vma_write_lock(next); downgrade = false; - else if (prev && (prev->vm_flags & VM_GROWSUP)) + } else if (prev && (prev->vm_flags & VM_GROWSUP)) { + vma_write_lock(prev); downgrade = false; - else + } else mmap_write_downgrade(mm); } From patchwork Mon Jan 9 20:53:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094292 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 517DBC5479D for ; Mon, 9 Jan 2023 20:54:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E52688E0009; Mon, 9 Jan 2023 15:54:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E29D08E0001; Mon, 9 Jan 2023 15:54:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCA9C8E0009; Mon, 9 Jan 2023 15:54:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BDDD58E0001 for ; Mon, 9 Jan 2023 15:54:49 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9AE3F120C1A for ; Mon, 9 Jan 2023 20:54:49 +0000 (UTC) X-FDA: 80336464698.16.9719947 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf03.hostedemail.com (Postfix) with ESMTP id F22FA20004 for ; Mon, 9 Jan 2023 20:54:47 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="k/y/f651"; spf=pass (imf03.hostedemail.com: domain of 3Fn-8YwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3Fn-8YwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297688; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L7MiPLggwHMt078/i5i4Bbk0h7mxL/B1TX4Loq0bJkA=; b=Yk0NDeTLgB4iLyiIdMEJhKcTvz78DGYoxmPGiW+Dib8k4jMhrcB0hpPI10bDhNUoVmgqdB jo8LH2vK17Y2MOufER6y3Z/qA/SAAB9R8vcvD287uSGhidacbvzUfOh580D153qQv7reYK kdj8qj8S+ETLeiAtwR+Oox9cNqvgBYY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="k/y/f651"; spf=pass (imf03.hostedemail.com: domain of 3Fn-8YwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3Fn-8YwYKCCsZbYLUINVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297688; a=rsa-sha256; cv=none; b=T8dgaEhyVu0tYeqYILcEDMu1BUvl/tU3699FLXQKOEP5PRWby+2NNcwjD7ZO6/8FdIo00y /b2ejjw2nA8TGMtpc2Urb65KB/Q9Ozb0TfQEAUqVmzyH5rbh/tf/oXsNJ1SxsbGtEK2+8p oe9Sm8VYmOrvpbDVXH/DwWSbgHouHrQ= Received: by mail-pf1-f202.google.com with SMTP id a18-20020a62bd12000000b0056e7b61ec78so4007808pff.17 for ; Mon, 09 Jan 2023 12:54:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=L7MiPLggwHMt078/i5i4Bbk0h7mxL/B1TX4Loq0bJkA=; b=k/y/f651ouIVm5VYXSBuN3WGA3r0siHAwbforN7cl7XldyaXSobuMoKnfRbHsUp2Zw mT4Fy24gz9zLv4V1qPdlGMDZmt4uQYDL7SMguN64hvgl+OPwnYaemI3hYYgdYfqYy2ZE RZpsDEbxomOp03K+XTMpAlOj5/FuHalix+oD7gvliXvyOvoSifjWwtoSfq337yDgZBWx NsM/wn5sH1zgbh2ZxdX6VDsXj4dH/jKa39D017jsbV+IQ14YXnxgSRRt8KTb8ppwgMfm uVAI77HN37Vy1v+4HMEL/4fAI+3c+Ki/ub6d+F+N4Xj5NK0ZJDRKlJXLfRFrsnczSTJ7 ObUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=L7MiPLggwHMt078/i5i4Bbk0h7mxL/B1TX4Loq0bJkA=; b=enyTlltH//NIYVziZ4BQtGjnJmlKqv0iX9aVWRfb9Rtn8rPZCOoUTC/f8vXIKduFx2 0jn5U/mbOYUTHacjgIQvVvUtghVs/ktpEvHwDGJhVg402HlNZfJH9q1T42YsJ10YZVPD fFzXyBVx1+q0StbmsSZFJbcyxNFKHIeWWfC5tKrOC+C1SazvvJOp40Up8WxzegSV1hvL in5E99vpWYHI9ez6cd4Tyda36i7tv7aMjbLpWZ+6v2WyPjg2/MEQINzMbXp7nanzmkSp yEfTyiZfyhSsWSoTTmW2CsrpE7zA5ycrtMckhBjGUnkNjHeCwBr7YIIjCsa7hTY2hjCy gvrQ== X-Gm-Message-State: AFqh2kr6FyFrwEeF5h+1BfRnsq4WCHOv5mGlPKgSmlGqgWJAUWixk3yN 44y/GzY8tvyuBKscRrkJbrBWX8k0Sko= X-Google-Smtp-Source: AMrXdXscxn+TZfW1mPWgpj2/8Haqh4tY58OgT8gYBU0unalxDUYH6+O0E4BOyMrkarY33KKQg8fQ3JxSbTo= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:902:8e88:b0:190:becc:7e76 with SMTP id bg8-20020a1709028e8800b00190becc7e76mr6134633plb.1.1673297686693; Mon, 09 Jan 2023 12:54:46 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:21 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-27-surenb@google.com> Subject: [PATCH 26/41] kernel/fork: assert no VMA readers during its destruction From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: F22FA20004 X-Rspam-User: X-Stat-Signature: 65rd8m5eeusccd5q76rr5hqdwjgy1xnh X-HE-Tag: 1673297687-264837 X-HE-Meta: U2FsdGVkX1/y8XmESI/UQTOuPK+Mw1EOaFJsGakweEPxHI5h3RUCtrlbUFcWxuExr3JO/VaM1l0YFV9dF+le40yBMQsq3pOwQhH9fg3HVodG0/hv3uRlntOEYbaSo4qRwf97MFn1cpvFZUta3ZKsJCtgi/gKIW+87oogiMPnGKpsQB/JaVnGPgZWKMHuHk5GqN+2xnwZsOHayzSYYzsRsptpBH9VqrsQohpA+b6RmtmMm2NPCcg187jzskTlL6kKLy7LgYxP6fKUx8uXsvYd2hQZFNpf4YrsN8+QRP8/mRvE19jIDL9wj8PLqR2pP+hb4srQuKaDnSZjH4bBOpIF4dPtsw3nmxErsTQY7fwKNg0rSnnZWCIMEzbRlB22yg6RkW9rFK8EerZvuHyr3QpGO0DBIAopwisthVqFuf+GPxF4jyWLC0CHIV4U13MnVdjEd/hUpcpAIOfzQ3X/YdDvl0h2veQ5fU0NN1BqmQWj1MASvkls1Zt5ub3JO03s6bq6wnydI9hwghlWCbchForScRuqrHUulbqAiE8p6eEpfJIWjBFDtPH4UBIgXMpitOb4ZouhfB6KuWSjrZjyV0gJ8Kj3A79TkqWoQM3xqKNNJ14fbVuNKerVO56UDcgzUp5bO0pqPyqe7gUV5ApNUCw25pn0q1H9aC9hRiLV/ARBct59MMtIwcBqoNbcQZr7wwij3nc7zi4P65LiTnoXoM80i5t9pHd76pSbRuPCvR5bRDCIi4U+FZLIDHXJdWjkIobkDMmNpKfyBr/sbMphLK2OgY0EaathoN33QOHHAidRreJxYVthsFmv98TeMFn/vGGqhwa52r1sMVsSsWNxs6RosWdeFRTCcv6984ncldZrrxyD6mCnBP8BZVftAhNRIEsksKfAnsJNIJCTq9EabJQImDOp9udY6OY1K6ZoZVmvSuwDZ5X7x4OmERmagEuBQhFIjye1U8XZtWbd4V1+pXr i9wXalft yloFVtS+dWKrA4aiFDxkNmRl/Rd4H7iJsC/YV4RdHxJm1pUl1BP/sRyhLVTrI+p0qDwfNVYTA237uTfurioA+e01u44+80pvEFr7VZ0o0QLr7Z1N4i7Vf2iEYkn8VoLkS0FnzldRO7ISBNVvyndyuwMXc6yvFbGsMom8uikpdqrs4YZAyRmPj+5QJZmJrOwPUelii/olTWC5Acj49ykomvW0pAz6hPJxrqyaEgWTgCyq9Kw8X+zgXuPxDo0FENM+XbdjUm+EU2v2SaVsR3BrNL0OVw6P7Hfgbd1qVpKUKSN4tWX/6I9F3d3v0MpqUTkHfnujHSem+Serm+bRkvnX9pq0RTDwJ5dhB0QRKhGsJFE2V/wA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000017, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Assert there are no holders of VMA lock for reading when it is about to be destroyed. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ kernel/fork.c | 2 ++ 2 files changed, 10 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 594e835bad9c..c464fc8a514c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -680,6 +680,13 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } +static inline void vma_assert_no_reader(struct vm_area_struct *vma) +{ + VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock) && + vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), + vma); +} + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} @@ -688,6 +695,7 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) { return false; } static inline void vma_read_unlock(struct vm_area_struct *vma) {} static inline void vma_assert_write_locked(struct vm_area_struct *vma) {} +static inline void vma_assert_no_reader(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ diff --git a/kernel/fork.c b/kernel/fork.c index 1591dd8a0745..6d9f14e55ecf 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -485,6 +485,8 @@ static void __vm_area_free(struct rcu_head *head) { struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); + /* The vma should either have no lock holders or be write-locked. */ + vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } #endif From patchwork Mon Jan 9 20:53:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 629A0C6379F for ; Mon, 9 Jan 2023 20:54:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0711C8E0016; Mon, 9 Jan 2023 15:54:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F3B448E0001; Mon, 9 Jan 2023 15:54:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E03F28E0016; Mon, 9 Jan 2023 15:54:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D13638E0001 for ; Mon, 9 Jan 2023 15:54:51 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9829580F38 for ; Mon, 9 Jan 2023 20:54:51 +0000 (UTC) X-FDA: 80336464782.27.BFE29C0 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf21.hostedemail.com (Postfix) with ESMTP id 042211C0009 for ; Mon, 9 Jan 2023 20:54:49 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aWg1ukrh; spf=pass (imf21.hostedemail.com: domain of 3GX-8YwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3GX-8YwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297690; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0pkQpuFsTWto6vflu1WV/u0ieoG+9lWN639GtltEg4c=; b=FfXrgCWiRhGx5oHr5yTcaglm4nKjcV28L6mR6BHmW6oSxzGNoE/GDTfklWBYszzHmEDSpX 2UUCvT6AIlJAaujx59ayHC/7m/iaACMJv7FxQXjNUhuylRpxVK4M+8FgMfCSiDSwo5of/E goSIXfGOVS1RhmKT8L6CfTXqErBbKyg= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=aWg1ukrh; spf=pass (imf21.hostedemail.com: domain of 3GX-8YwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3GX-8YwYKCC4cebOXLQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297690; a=rsa-sha256; cv=none; b=0rr2a6y0Go9aqeKpGxrW+G6199HhH5p2nIz+ws7H96kT95NZ+QStZmH7OCSJskCuYBpYjc cBK0N1OHNO032qWqWv2o33wBFd/VggT/A2RDT8vRrAP/O5NcH/Si6PVzvuUEQjpU66Pp4J D8Qw6OqgMDj06uOVrf6V6twbEi12Izg= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-4c11ae6ab25so104150127b3.8 for ; Mon, 09 Jan 2023 12:54:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0pkQpuFsTWto6vflu1WV/u0ieoG+9lWN639GtltEg4c=; b=aWg1ukrhEhZM2G7oua95ritICYN4v22wUCcY0gEG4tVylqBmv829owF41k3CxkyAaw SuqbgaxpvQ3jo7YPBWRSVqYiX7AqAmWKt0XHBKUG5SuIsHysEKVzXAcfr0KHgTcUUWfl b8z+Kg1PDae7rSd6mmxN8sCBWlq1F+xCCZRkBLC5iLo/l2X3T6+e58rjDwLcesFby2PU TDwKsOLIZU7rmZo8L2TvumA953Nmin0x2oWW50UYSkBp0m/8PkZaL+q137UQ5Jg9l0HG VtNsJ9z3uJiIVmCVloCQiu4j8+XoGxPmrLqPQDbiOLf/gZJpQPJ4VggKimmoURCteWUt n6qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0pkQpuFsTWto6vflu1WV/u0ieoG+9lWN639GtltEg4c=; b=ZHd9TJ6OQFORXTfRy5RQQAun65cWr3eM6WctxBI77h71INFdSoExV4eMsQJsgSDbo3 c+1TzQ2gRaNyUv823aOPKQp0V1n9R6iR7qcVSTQvJrs0aMDI/LGyDTiZkfHxIZRUFvW7 wY20P9lFG6lx5ieNnZvDCZSTTStr/NdyZB3fiMpzWYXt+v7ATkrc9EZjABKDDxGSJubz v+Z7qyDF9h/ZRcxf4CKeiO5n+xUGWAhz93O28sFnkrB130hQ1Y2CJNfDuBpifrgrAuW8 4f6M2tQtrDyPnDm7Hde0YcdNyi1xyu87iBoM+zBGD3F/cw3APj5oR0WoRGHXblJHmSPd 3QxA== X-Gm-Message-State: AFqh2kqMCKjNx1Z40cqgkKuDZpxUsT9S9yB1h7ysCZGlPdEY5tU0Ab+P I9n6HKBnhCeiGSADP/4oMtGV40/VLQU= X-Google-Smtp-Source: AMrXdXvMFgW5AuRJO7w2AGyv4a3g+CMQq4rAqpoqS9oQdpJUn0mmZFftW7BbnOBV6/f8EJVvUFaCfl4rQG0= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a81:e93:0:b0:3d6:233b:7292 with SMTP id 141-20020a810e93000000b003d6233b7292mr1998513ywo.476.1673297689205; Mon, 09 Jan 2023 12:54:49 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:22 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-28-surenb@google.com> Subject: [PATCH 27/41] mm/mmap: prevent pagefault handler from racing with mmu_notifier registration From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 042211C0009 X-Stat-Signature: jrfj6uopetfkbsj1fnj1dgsfrnji7pgw X-Rspam-User: X-HE-Tag: 1673297689-6229 X-HE-Meta: U2FsdGVkX1+pYtSDMD/K9m/7F82prhkloXYoVMizZ4KAjeHb3izZhNTvwcQEOpUayutuePwcnf8T9b1Ngc3VlrkyIG3O6YeaPy6A1S1ayldwbgHaJPLvXpyGTwjd+BsJsf4IibvOTx3d0dIjeFO+F7nUSDrpBiiddtlaLLf/YQmeSgsBFOu9WcUcwll60meVBBQk6uz8c5ZoaqOSf3Y4KWM2t9j/mI5SfSd5EjX9Cg90Oo/t/eSldnqrEXj5nT0KOiWqroAaZjyTHIJcyNceubw9Zrq6nX8MvvAgQ+U142QTFdCVhIDoX3egVFsH28hCqrccWiUg4Z/RCGUnc1MERvr0MMoMVBUiuLanCf5nyMcX1/4qkY1dzoQ1Uemk07OGFxg+rXez2PgH7+70mTdkBvmIqFSW9df5quioo7/84F4n5tDVGLqikBvirvgVeLtyY/9WQssh6RMLAl6TOrz25kN7lmIA/E28bjaIDtLFING+ydSUtZsa7hMmyBKj1zapHS8Hty4n80jo0mSvEBEgTfAdxg4ZoVhXmwrJvqre+CKZwa5AAGISAn73HYltMgzFVmzgbEp4/OWtFCwXxvB+I+6yI0ZWNf0gs+gFWTGKJWekZUNyKkCbzuwBpncPRuc4tDj3FDW1MiTClSNBkSGciR7cIaYeLikY7IZ1IjePrlPt7qYpYYsm8VhhkezMORsO2cdHKQzByDKCwl7ymjhEVcgIikLwTiFhxG0mMuEhnBAV8llCroHFsSElfqrzrUvmFUZnrUK1VNg31aO999fslIOTW0jcphlb30uK3+X7qz/OxSuOWc+Yny1FBcvXr9EQCq4qZf2SYKsrlL9gRmEwu05A2/i5SQFAUHtW/BfC45G12NOCIAHXgjOktyr2um9tjGlUyVYiotpAHrModjMspOlXAVC+h5UfCkpoI0qP3Nw1+V9aNc65ypsHflZNfcM5EnggexfeE+/i6d6OBXl rhxcNTJJ lSO+NdmcndjgmKj+xnv9EeMga1C9mzsi3MrylEy9G3NPgUiPGfDsqVO9oJOjYpHqOpS9YFO1g4qoB2v8wN61vF3kyDLfoX1wFZaEHrC7b2FpLNWg1Qxl6A48ruX7UVdJa/tE9RvC78xlI328dQeuVSCo5hWkPiEz4Go/k8GFabSXD+bePPIQGLlTCsnUlHP/fKottNGPnRTxiKdDceqjURzGY1CrngNm7JMv4DQFc0iGqfdnHrObrRcXpIsgBFAZ4+VZ7lrXhZreRKVZd04Q6poCHChllSO4LNq6EnqWDXdBF5G3m+yz0az6Lx0XFq6U4N81VwWEPOY39n6KoReKB/7L9YVGnoovpyjLSFiPpluRxFXskd13tKFDM43sqPUqXAtH9dCz7p+b6W53qs1ZEHvnbmnbFtNEDSVGf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000007, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Page fault handlers might need to fire MMU notifications while a new notifier is being registered. Modify mm_take_all_locks to write-lock all VMAs and prevent this race with fault handlers that would hold VMA locks. VMAs are locked before i_mmap_rwsem and anon_vma to keep the same locking order as in page fault handlers. Signed-off-by: Suren Baghdasaryan --- mm/mmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/mmap.c b/mm/mmap.c index 30c7d1c5206e..a256deca0bc0 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3566,6 +3566,7 @@ static void vm_lock_mapping(struct mm_struct *mm, struct address_space *mapping) * of mm/rmap.c: * - all hugetlbfs_i_mmap_rwsem_key locks (aka mapping->i_mmap_rwsem for * hugetlb mapping); + * - all vmas marked locked * - all i_mmap_rwsem locks; * - all anon_vma->rwseml * @@ -3591,6 +3592,7 @@ int mm_take_all_locks(struct mm_struct *mm) mas_for_each(&mas, vma, ULONG_MAX) { if (signal_pending(current)) goto out_unlock; + vma_write_lock(vma); if (vma->vm_file && vma->vm_file->f_mapping && is_vm_hugetlb_page(vma)) vm_lock_mapping(mm, vma->vm_file->f_mapping); @@ -3677,6 +3679,7 @@ void mm_drop_all_locks(struct mm_struct *mm) if (vma->vm_file && vma->vm_file->f_mapping) vm_unlock_mapping(vma->vm_file->f_mapping); } + vma_write_unlock_mm(mm); mutex_unlock(&mm_all_locks_mutex); } From patchwork Mon Jan 9 20:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094294 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADF8FC5479D for ; Mon, 9 Jan 2023 20:54:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DCC98E0003; Mon, 9 Jan 2023 15:54:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48CFC8E0001; Mon, 9 Jan 2023 15:54:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 32E968E0003; Mon, 9 Jan 2023 15:54:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 23F548E0001 for ; Mon, 9 Jan 2023 15:54:54 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 03F10C0F33 for ; Mon, 9 Jan 2023 20:54:53 +0000 (UTC) X-FDA: 80336464908.14.AA328D0 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf26.hostedemail.com (Postfix) with ESMTP id 5EC1014000B for ; Mon, 9 Jan 2023 20:54:52 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=T55Sv4TW; spf=pass (imf26.hostedemail.com: domain of 3G3-8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3G3-8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297692; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UA5+XYk5SNGwY0wBqjbKeTB68Smdf1b2WfFGnAueiFI=; b=aLSKYTjSglZhgpp2YZut65CdLFpYZl5+h1A4PBRiMPu7Lg2aO7gVyQ3Yv4KjlLEyc5wO3V sOMSriOA/UZ8ncj4kQ00nxgq4b1/5wTm5zWC2l1CIibAI3KZpJTGSh2HS82rUpmu5gYNcl J2VfPGMiydr+YPVM6KYCwIppwbXeNJ0= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=T55Sv4TW; spf=pass (imf26.hostedemail.com: domain of 3G3-8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3G3-8YwYKCDAegdQZNSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297692; a=rsa-sha256; cv=none; b=e8uwJ+eHyDewmbvHPPrHeS1t7g8KuMX76Ma/KId6rILkOrEVUD8wjo1dUA9wCcQgMbVaW3 BGFVrJNAaZIpXCey8FJ40iGxVz+p4JUKbz0jeHBpUKgZyg49mr8b0czQqwdSEnPyQUveHn jC4eNCReAj0/hsvQKxE7+9QCWihCCz8= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-349423f04dbso105151677b3.13 for ; Mon, 09 Jan 2023 12:54:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UA5+XYk5SNGwY0wBqjbKeTB68Smdf1b2WfFGnAueiFI=; b=T55Sv4TWznVzBLEvpmky/jAR0QJoeEeCX90o+hjvZVdyHXxi4X+dLVqm78fwsxY8qM 7mstQqJ2q4Sn5cjstBMo6qt6HReHOpVXqnm5LMKSQ635M0MCSPIfIgxAkOTHc8oB12I/ ai6kXEsVt00A/xCygYMywSMRtpTqMTCmAJNGp6/HS0isXJekWM6FOqzBH2+trv87L4rS u+h7CTRZzB07Ow9ioaR2euNV+QdGF1fgPh0hG8fHh3bLFfxrE2kcC3qaflJTUkk0dQTh 8vb62xpka2Ra5fG5NjEzQJYLTJMKCpwH3JBwVijZjrLZOBQ13qNu1ZZEA2iHn/6uZCd9 oE+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UA5+XYk5SNGwY0wBqjbKeTB68Smdf1b2WfFGnAueiFI=; b=gtSBro3W5+d/vhM7vwE77iFChU1Q30Kq2/jC4NxWZW1dPQOPfgubeqcghZlHSf5V1v 5pFtheW5qw8O1QrkxEISv1BTQPyf7n/JWovMU5gle1FBJCygHNoYhqCz3SF/CbhKpmgM 2VQD8vsEzUAon1cZK+ONB4jK+c5FrwhKHC9zafyrKA5ZFU9Ius8d6OIS4DLRhM2rvPVp tXaztTDw+2KznuT37BFqSZREsbxwbEuHQ/ICtobM8wgrXyJ6mSElc++aWrq7zOIwlymA V0orjC90cmmo+Fwc7Hvy4Ql+IFHFnAsEF/rCQE+7QkwosxL00INEbBEr+s7nKU2id3me QDrw== X-Gm-Message-State: AFqh2ko/Fuol29KA6OdvV/LiDmAEZyDnRS39lMKDlApPGlgrutHEociZ dBoNXqcCDbyMpUEfWDbB8DMtA3dnqS0= X-Google-Smtp-Source: AMrXdXsg8SKnvt1lG56Fk9uUJaowgWL0vbqBlxMLAmQhblw+uvVMcBKNRkD6OuTHfHWzX2ebIh14gNZU+Cs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a05:6902:920:b0:781:eeed:9abf with SMTP id bu32-20020a056902092000b00781eeed9abfmr7003326ybb.538.1673297691543; Mon, 09 Jan 2023 12:54:51 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:23 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-29-surenb@google.com> Subject: [PATCH 28/41] mm: introduce lock_vma_under_rcu to be used from arch-specific code From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5EC1014000B X-Stat-Signature: s731aoyy3z3xzhbnh5r946hr3ptszjzs X-HE-Tag: 1673297692-566057 X-HE-Meta: U2FsdGVkX1/X4m6Vm83b6ozKcdKQCqL5MpVtBjOM2du/ykWsRDGw9KqXDFO/hkf1oBGDLVm2F/xitLpsR/xBAwc+1BT1B9z2mdismQokGrf6c0MtYWejQMhqLw1JpmqJRf0iZ3MrSlnuQTO0rkVfpskPg0A1+7m65UcyVUPblFM3ptFZBKtPOKzNaycKfqHKOEpc33k2x8jfxjP9Hven6uFsIitTfJitzJ60fxSsQqDpWPHzSqNGx8BHm4BXWlBz3qFzMsJxL7emDUAJeD9roPgvw3UUign/Ay2CJcvL3GQvDy0J6drZeZpP3tNPj2Vc6XrpdZNVj1nxxiaJKxscUXj8VeTjevBAhLnP4Jq5/q/OjO/9WIzpj/wCE4nby8JhUkwXxFLhhWcV1s8mXbegjMMv6p78rs5LA02q8qlYWRrERxty5tv438eCB1gD4/jTwdTPH/cT+kpJyOGy2gIyOfob6Fexk/E4FcRg9w3Ta1qmbgtv9HBRn98LMsoHSNnCdc4l/fO8xHczfTYZyCbtYh734E+okiOOyp3/Rbx9+0gl7if45ep513mkn2MAnCUMzeopmB9eJfMo+FES2giWH3OJXdvBDKKxC8uUlD2/jJcUqHA2wwsvc4zN0StHyuysXSJh2iOglURJmtRvuf2rSzFJgc3yGxUu5jcpUBi87tY0xB5pI/xBzjjlGY9rATW1K1NLH0QIydKz+RLA13YcDINsbuRmp+z2ruLbQrnSTREJRXtgiQKoI/Mzhc3PSauag88LQ3sOPdq8Iyi1MYp6jjtbbQq2XFPh6BbPwn29KrGXyWt//gtf9ubecLnIYJo25tzvi9FuBt2Ykr8/UR0b1rBb6twJGhpm4tN0YpeX6LDBZ8PNLWNbqOD7Bvzwp2kQQuR9pEj/GpP0wr33fFwG8H3ubuygE9kDq8jqyLyn/kFB+K9ztGCu83tWcJQXWoLa3sf7WgTBIIj8Gb4F1qF LHMAKBbi jHDeJI0sMdDuQHaV65JvLEe2pNLrxo2kfPUrlw7OMz28mz+TCfOd0wb6Q3Eqc+b+G+Gea/ESCrxfVBH+22wrnY+CtDQQV40HSjJfh5VOvx3wmBpyv+HrwO/T5MZ8i+ftBN/r9lN/cX6m6dtSrxpMopCt32pg2WU75g1Zeaew11QD/anqcIHy7522zZHY+uX4uGez43pgx2JoHPDDZfIzlEafueFhGzS/Q98vVi/OaIaoZoAYV6SiV7s0NSZtVfN/OAdFXFDWhPUIeQK2oxnc6PVON2GsJ5LqelVOhbE6pPpClF3yPlA1hie7iEC3G+2iBbsmUhS5xvj4Atl++iGMKBWJWK5BkQR96GL39t2dDa1bSAW6pYNGKUzxzcoQaFIljM1tKS35CfWHjJgS2PHFK6wk6JcqB3WACcXnF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Introduce lock_vma_under_rcu function to lookup and lock a VMA during page fault handling. When VMA is not found, can't be locked or changes after being locked, the function returns NULL. The lookup is performed under RCU protection to prevent the found VMA from being destroyed before the VMA lock is acquired. VMA lock statistics are updated according to the results. For now only anonymous VMAs can be searched this way. In other cases the function returns NULL. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 3 +++ mm/memory.c | 51 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index c464fc8a514c..d0fddf6a1de9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -687,6 +687,9 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) vma); } +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address); + #else /* CONFIG_PER_VMA_LOCK */ static inline void vma_init_lock(struct vm_area_struct *vma) {} diff --git a/mm/memory.c b/mm/memory.c index 9ece18548db1..a658e26d965d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5242,6 +5242,57 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, } EXPORT_SYMBOL_GPL(handle_mm_fault); +#ifdef CONFIG_PER_VMA_LOCK +/* + * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be + * stable and not isolated. If the VMA is not found or is being modified the + * function returns NULL. + */ +struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, + unsigned long address) +{ + MA_STATE(mas, &mm->mm_mt, address, address); + struct vm_area_struct *vma, *validate; + + rcu_read_lock(); + vma = mas_walk(&mas); +retry: + if (!vma) + goto inval; + + /* Only anonymous vmas are supported for now */ + if (!vma_is_anonymous(vma)) + goto inval; + + if (!vma_read_trylock(vma)) + goto inval; + + /* Check since vm_start/vm_end might change before we lock the VMA */ + if (unlikely(address < vma->vm_start || address >= vma->vm_end)) { + vma_read_unlock(vma); + goto inval; + } + + /* Check if the VMA got isolated after we found it */ + mas.index = address; + validate = mas_walk(&mas); + if (validate != vma) { + vma_read_unlock(vma); + count_vm_vma_lock_event(VMA_LOCK_MISS); + /* The area was replaced with another one. */ + vma = validate; + goto retry; + } + + rcu_read_unlock(); + return vma; +inval: + rcu_read_unlock(); + count_vm_vma_lock_event(VMA_LOCK_ABORT); + return NULL; +} +#endif /* CONFIG_PER_VMA_LOCK */ + #ifndef __PAGETABLE_P4D_FOLDED /* * Allocate p4d page table. From patchwork Mon Jan 9 20:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094295 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53D8AC61DB3 for ; Mon, 9 Jan 2023 20:54:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E52B28E0006; Mon, 9 Jan 2023 15:54:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E2A0C8E0001; Mon, 9 Jan 2023 15:54:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7CA38E0006; Mon, 9 Jan 2023 15:54:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B69A78E0001 for ; Mon, 9 Jan 2023 15:54:56 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 867D2160F3D for ; Mon, 9 Jan 2023 20:54:56 +0000 (UTC) X-FDA: 80336464992.26.7E0A0DA Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf08.hostedemail.com (Postfix) with ESMTP id E4B8D160008 for ; Mon, 9 Jan 2023 20:54:54 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GjzlXD65; spf=pass (imf08.hostedemail.com: domain of 3Hn-8YwYKCDMhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Hn-8YwYKCDMhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297694; a=rsa-sha256; cv=none; b=Pc/JCzebbo+rvmZwJvJCFNEZ6uv615S7ZpFienUJk4eKjTv0BBmwzcb+FvxCoCz8V9zjz7 qeLMVfcNPmLsL6TGBYE4M3E2iR/k2Lt86rSzzMm3CMK6L5dj+Af6LLo8Z/i2fxHzUUGBuy ZdtsBbJd41KxLPsE/LBWVFSs9ZC0S6U= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=GjzlXD65; spf=pass (imf08.hostedemail.com: domain of 3Hn-8YwYKCDMhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Hn-8YwYKCDMhjgTcQVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297694; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bx36nWIWIaMqsyKC4DFh9Dy6zgkZnlvdwV69ikS09tQ=; b=AJM2cV5qxtsHKHZ5Yo1hiKCAocTstdnrWRlcg6Cumen9qrDJR1CSurBiUsQsdqTvhUpoAT Y3rqEZw7xO8fFOjkT28yE7n9gjYc6YYWvhFOoIAa7q6OCTy4GRCx/etv5XPWvrPpE0F2lW JqAdkEZ3K0Vq6TKl0lhcDijspTQyNMc= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4ad7a1bd6f4so105783277b3.21 for ; Mon, 09 Jan 2023 12:54:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Bx36nWIWIaMqsyKC4DFh9Dy6zgkZnlvdwV69ikS09tQ=; b=GjzlXD654bxHD8qHg+BCMZfuuyb19mHGKgBfyrdYtuHPZI3RXY2mylK8MrrZ6vi/KV XpBGPAViKdrJ111khQK0LarqUyqdysojbfZLW54fo2JQeFn5ElZgOwEGNBdof4Exx32h oce1agjXuGHjstOPH9icNCzJQUnGWQYtYgBGU792WEzYIKYlpSXiCOhWF2Ne+MX/DqXf xrifjQ0boHWSv5tXEGAlsRbW4dbyfRz7jLqAVJ0lSkujXDjZ2O7m0HOdnuYL7/iH8Jk5 Up4mHJat+wejXPO4fCLDWlYHvfZ/+JBFjf5EhWbTUQtFaZy2JybPs/XeOQ27EyoIyzsK wpvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Bx36nWIWIaMqsyKC4DFh9Dy6zgkZnlvdwV69ikS09tQ=; b=2iKXRT/m4ezNLhnqJ5ByLP+c0RKGAoANSESMZ5zQxbBS3fu5LgIwaPTQcfkRoJkRwV VlrXMfo3kRbLWsUh39Onw/AeKw8JbVHpQNbM0wVegXahx7JPEgXVfa0nBVGKvaXAGPyt NCzXMjDtULCUYLQ0f52CK7HGPa9RVuosi2qiCailx+Ylv5AwaawZZatQ10oXbB7J1LnF 9qyKa4MtcUpzv5gDeK48fiz5usOHmgvVF7PCq25mYbjLSSsnA2DPpnJ5PsjBMCSLP+3V X/LpRNSjXhL7TmP3diDadvX/jYPJxu1CsKyGAoJxzMmSAhpQttDgUc2G6XSgni8lR5Zx xZ8Q== X-Gm-Message-State: AFqh2kp77klHqELyFv4hJcfiqSgs/5OyakwKSsjygvDn41jhVUzNRuIp LhslSCtjLMLhCh3Nhmmr5D5Uj/Gx/7Q= X-Google-Smtp-Source: AMrXdXsUnoeDnhtdWoii8DMH8rCFUM9XgPs0a9yaOqp3Qv/VXJUnVpuxBD4Is/HyplWdMowbocjku3a0aLg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:3c83:0:b0:700:604:2e1b with SMTP id j125-20020a253c83000000b0070006042e1bmr7014317yba.246.1673297694067; Mon, 09 Jan 2023 12:54:54 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:24 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-30-surenb@google.com> Subject: [PATCH 29/41] mm: fall back to mmap_lock if vma->anon_vma is not yet set From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: E4B8D160008 X-Stat-Signature: okyiss6irep1gcipciuyzfm8sd7yq3c4 X-HE-Tag: 1673297694-900668 X-HE-Meta: U2FsdGVkX1/mbpzFAEcTWIjAlFQFr0pMylGab5euLjpHNunZvupq9/ue1yWvvzLiqckqzRftadCVKR8Kxxh4COiTTlOWBZ5R/eSeJneKvL3ReKVYDAU9CZQOjxnqkLc4pmQHvpNi8MLqLF9NBmRS7fiQQsfZP+CkhpCkOxLDfSnmtvshU3N8mqUAW474+xF7dD9JSnXYJ7pi7bnXH2wVanJZn+/HA0fFzjdtnoe+D7b4BX6Qg1bePIItEXNn9w3vfNAKwlWY5/CqqRXAcxjc4CTd7lLbDClaAYBTSHXid7krhYs81SxYQAeYbZgRc54aSUt0VbyqaC9tLEZ+b41zL2dVyHHA1payUfiyxOfWgMuBtCijrGJMVvDc2bQ1G1Fvyi/Z8A3PIweoHKsTFTQwJ/5KQo1YFXmqY5eUnWMyFSWkIOOrcA204oNXGDOWEQll4v5Uifs88T81+BNbwUYk53jw7Rdq/gP6iosp7CfiLc90uq9nomzzmR0ev5XYo1OwT8kPU+HTW+43bD6lh5wyyas0dwpSHWxQk418+Ls/ZTt8M/XZCcPot/5BG/Gu9rV89pgaOPjzKp6kOcIRHpAossn87GKClZBMFqBlKi4Jew3c1gcMOSktpYirifpG8NL0vURiQ8RaTL9GfRIgeEHaTOeTzASB0sTsexXID/fM1GfGj+6Q+mYViZFMWR7Z8qPBhIrI9yw22FC+irHqKXPI44Q30lPJUXZYmZt/YO1Fq18gSKzvr0fFGa2WRTco89FMJ9CJwYQjZTL50WnoNKrMnxzAYCLqK0SDdgkgwhAtowUO55A8ASlbIMzaY649YvPYkwXDKfYXI0d8bpRq/6SQaFD6bfPXrOOQKJ0ZtQwi/ruzLMFmR3Im8vGfIv+9wpaSXhHNRV7DCyO4P38Ud90TasiiYSsjrecyJkL6Tli5BVBaKIvKTb5CA3NBb22CwUCqASTeW6aYkjxJH97byvk MEGjPP4G x1s+WNY6QHGkrr3+Qt8RNZmtC2G6ObqBuV7gqfaBKNr5RM/Z79hM+eF4vlKhGSE8gWj6x+sNXiGM5fx/i3IFSJhvJ/NrYLvisHPyyKC+lPqkqIFp68VfAhwWePYrVikI3mi2UedVn1mLMF45xusxJ1ej+Z4N9aJfefY9HsUvNPzaVPWdD6sS8oA3E4gF2IGw3OTneFyhecLXXisJCZ3m0DupJly2XlERt+uiINPTsiTuzesQ/Cux+cZ2Cj3P8GZMlwo/wsQN9QiIwHplb3j9c0wgHLy9FamD50X5rqJk015CI4lQjNbq+K3DmR+N59+SEHfMMoieTkRSF1Pas1UXfQQmaYLwgb2AnBjunoUsM2EuOwE1oa0PGzWeDQNANf65wuEm59Jn/gxz/zLmcq11FTPc7EhhFwNoRR2g3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When vma->anon_vma is not set, pagefault handler will set it by either reusing anon_vma of an adjacent VMA if VMAs are compatible or by allocating a new one. find_mergeable_anon_vma() walks VMA tree to find a compatible adjacent VMA and that requires not only the faulting VMA to be stable but also the tree structure and other VMAs inside that tree. Therefore locking just the faulting VMA is not enough for this search. Fall back to taking mmap_lock when vma->anon_vma is not set. This situation happens only on the first page fault and should not affect overall performance. Signed-off-by: Suren Baghdasaryan --- mm/memory.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index a658e26d965d..2560524ad7f4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5264,6 +5264,10 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma_is_anonymous(vma)) goto inval; + /* find_mergeable_anon_vma uses adjacent vmas which are not locked */ + if (!vma->anon_vma) + goto inval; + if (!vma_read_trylock(vma)) goto inval; From patchwork Mon Jan 9 20:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094296 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 876D6C5479D for ; Mon, 9 Jan 2023 20:54:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2934B8E0009; Mon, 9 Jan 2023 15:54:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2445C8E0001; Mon, 9 Jan 2023 15:54:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E39C8E0009; Mon, 9 Jan 2023 15:54:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 017648E0001 for ; Mon, 9 Jan 2023 15:54:58 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id CF9491C65BD for ; Mon, 9 Jan 2023 20:54:58 +0000 (UTC) X-FDA: 80336465076.14.D2C9A52 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf05.hostedemail.com (Postfix) with ESMTP id 352CF100002 for ; Mon, 9 Jan 2023 20:54:56 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=RmMHlCmm; spf=pass (imf05.hostedemail.com: domain of 3IH-8YwYKCDUjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3IH-8YwYKCDUjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297697; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B6J4oGWYAsMb5e7sz4HwYEXURXCKE081WlUNIaNvLis=; b=XnWLAvUZMfRO+tcBYMoUYvyrUU4zYbISk8HJypL1gBoaYKHmQHodZ49nrsebqnq20RU9v7 Gz4uBlYTBE+8HdaKY2nSh84RUdUGSZNq149MpSMzDT3iyP4M+3CHQtxsXs5m9JtdVob0Ew eIJ7sEddtQrx5PGIf7zDNa34zf7hFDA= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=RmMHlCmm; spf=pass (imf05.hostedemail.com: domain of 3IH-8YwYKCDUjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3IH-8YwYKCDUjliVeSXffXcV.TfdcZelo-ddbmRTb.fiX@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297697; a=rsa-sha256; cv=none; b=Re+ctcaVwUTqB2lEH6DkcxLLuPLdvhgHYV3uDR9JeVq6bOf1uQMgvCSVw7kATRelyBCLIB VM/snaL4rSvXE6z7UYARLMhcNYoxjypkGEV9nILLdnLPM7VyfJQPAT6pEIDyV/U65PRAHX 64rZ7ACH4Bg+7fnDFx3/stQL1VBImEY= Received: by mail-yb1-f201.google.com with SMTP id i10-20020a25f20a000000b006ea4f43c0ddso10382392ybe.21 for ; Mon, 09 Jan 2023 12:54:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=B6J4oGWYAsMb5e7sz4HwYEXURXCKE081WlUNIaNvLis=; b=RmMHlCmm38xVKeEsK8n8bfn/aY3rbZqU5QjWJwlg7PYpTvR8g1QwX79gxnClyI+zIW JCQh5e4SjFQ2Q8nxF5bjIKp19Ks1VtVZP+kR4i8xND+6Fj+6CTJ8vlU6n1mfUhVhmmz7 +sE7byDQX+/dNdDepWb8tj8CSiQt6RCyFFAm480IRiAV2JLM0c7a5e0c36lCq09bYHwP wg+XvRL1iHdJUlNZBka6/ieT4oXpnPfG2/UuJktrbb0wvhMWj6jl5RcTqQ78qy0Rsk8c 4BQSRlXhCecUdC5ZETwJhxsGpmEcobjcEL9EE3pAQY+ImA2TDfMr31CaGMQDdx5w7rZe dkaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=B6J4oGWYAsMb5e7sz4HwYEXURXCKE081WlUNIaNvLis=; b=3eUY457JqJqoCTI2OTel+oiMCbcLDzSC1WiHarWKU5Q/gvKxguhsgQGoJVYH9Iy9x8 K6iWXxTnhS+R7unEXkIl1rAYuOOhYQoSDF5lB7SZprzV7DiaNKEGH/mWV+a0HlRmzpDe wbQUZPdrMmaDCPpXFAROE7Ke5FWCwPx+jTY+YUWMiZRmlIxqXMEFkKhkH9gFXk1tEXSx gExuZwmMZnbd+drsLBPQSs41myHKSeGO++8P21sRr0XYIEgt4b3CdNmP1L+ILgTqG0OC ChLRn4YmowlR54r2iNoi48l/p926GbY3QXs2WT9ZE1yni8s50T6zu+ixKfVU1OSo8TGi nQ5g== X-Gm-Message-State: AFqh2krTSYYgdc63atLKdgEOb1vftYWfdRTkE26JSqRQYhnV00ulEwe+ Kw2aOoc7nuAnQ43hkhOdbgyLMpvqZvM= X-Google-Smtp-Source: AMrXdXvwV2DJ0LbEXE5ZgdwVnwlID+Hh7Y6icoi6WDsC3E+JAvvNZYvjiQv0jw9VN21R/xnRr9wwO4IvIuA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:e7c7:0:b0:7bf:cf52:ede5 with SMTP id e190-20020a25e7c7000000b007bfcf52ede5mr196156ybh.626.1673297696306; Mon, 09 Jan 2023 12:54:56 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:25 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-31-surenb@google.com> Subject: [PATCH 30/41] mm: add FAULT_FLAG_VMA_LOCK flag From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 352CF100002 X-Stat-Signature: j3pjtbw833ducebgefzpk83sqde7fzzb X-Rspam-User: X-HE-Tag: 1673297696-335273 X-HE-Meta: U2FsdGVkX1/RZPcPAmmkS83IrjBlppd17udKNxJpjnB9Xgvylr/7vo1Cl9nBNanqe+68KEpTDPndTQ4iyWj9TlVp0+w45HNxcH94Ua8SWhTk7eX30cswqzMmaAGedBqgWcsTLFNZ2mVhmebtrY7hASJjCJQfCB5WqSJfKFeYfd4hNiEZiRe/NhjvQCW5oVvq0sEV3C0wqFCT+cFMyjUmbzBeD1QIT2BtDcNK+hIBH4/xxhLJEk/n/fO8nT+Q9CqfHJmnxlstq+JbtaDRDzmLRyxao3ckSyqzL4kpdD343p5v6UwHwj9AYUIhaECGEqeOlprAMW77ITB5BAq3vWUgvDhXWumO0amHgNSlV1tidnl7YY2m1OiGH4tVnHDii4WLP+4tX5115tsaGHF/vagBe0i2ftT+HC9BOHIOlAqX88wQ2JGH7+W8HT1bQ+n2fAVDkW5/OS8BYrooVH+5FlptArghtPs4XGz8PqZ0RYIYjFIL0zfRWG7xh0+cvemYfTOlOWSj8Bh5ERz/wo8s0NyldpKCoZAZ8PcBF2qdRsjbjRdmMdxzq77hLKVDJ7i49TS4J8AFzQduRFKUv0ACfaAEeo/jUZoB4a3/Lu2hA+w158/QXZDAM0jHZaxU9/IRas8fpAsfthfOHXek+VYasxp5ANcZ7wrT6+JEoZBYjEtMYrNPW8dOHPXZZBB6wgx7S1in2gfKH7b5uv+w8fhf0VqpR9mKjmtj4lZA7O3rMvgidNVlCjarvDthMwuMd6WfZjIt88jPTee9OZgJqSdGZKjxnMS3ihFHObw0NjsRzuIN4n7F4zeNJkjxQRfe10cW392drgwr+DOqGGsN6Y2gbUru2iNWffjr4/zL0SHiTtJwW5fh1xjVs0T6If6QdxtxLDrz1vcAoU3SUahZ8UdNvDm/eAZAZqjRZc16FgkLt1mEecmW+FaXRJd+cVey928mPePVW/mmJK+ikDFry1hFWux aG4aqXni 65Y3IURTh9JF/+XuREo8sPx2Ok+svorkop3BG99qunkTP5RLJ1XvHgwXq7pbHCpJfu7OAwO7FdbBAQO9gf7nenkty86QaRFGsLk5Omgd4f44GvgkD1QbzXXaXwjoPW371VitAKv0do/ShnNgvQYy9sz3BhMxZbf0rGEGood66nhjsT2R4MiENm+hTpdHnROWJ6ZLfchIFX/758UyViOLIvVBnfHfNTijwH9UQV+z0TJcM7HzqWfNgYMoGu0s71k5iSF20L1K0Y5D9vOwcNWNFVU69C0AxKwsQykBoosdmZp6AUxpFeflyo5ec+b1SnUbYv3KOlodX2/lTvYeVC8rWQkU6C4UkVF4c+ZGMVOqJFDuOP1Tchw2DUruzgdFfWxNiX8oXns1O5zDameU6CCmOUD0Y+JV+HBYbWX8X X-Bogosity: Ham, tests=bogofilter, spamicity=0.000815, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new flag to distinguish page faults handled under protection of per-vma lock. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- include/linux/mm.h | 3 ++- include/linux/mm_types.h | 1 + 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d0fddf6a1de9..2e3be1d45371 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -467,7 +467,8 @@ static inline bool fault_flag_allow_retry_first(enum fault_flag flags) { FAULT_FLAG_USER, "USER" }, \ { FAULT_FLAG_REMOTE, "REMOTE" }, \ { FAULT_FLAG_INSTRUCTION, "INSTRUCTION" }, \ - { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" } + { FAULT_FLAG_INTERRUPTIBLE, "INTERRUPTIBLE" }, \ + { FAULT_FLAG_VMA_LOCK, "VMA_LOCK" } /* * vm_fault is filled by the pagefault handler and passed to the vma's diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0d27edd3e63a..fce9113d979c 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1103,6 +1103,7 @@ enum fault_flag { FAULT_FLAG_INTERRUPTIBLE = 1 << 9, FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, + FAULT_FLAG_VMA_LOCK = 1 << 12, }; typedef unsigned int __bitwise zap_flags_t; From patchwork Mon Jan 9 20:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8E95C61DB3 for ; Mon, 9 Jan 2023 20:55:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 63BE78E000B; Mon, 9 Jan 2023 15:55:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EC688E0001; Mon, 9 Jan 2023 15:55:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48D2E8E000B; Mon, 9 Jan 2023 15:55:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3BBB98E0001 for ; Mon, 9 Jan 2023 15:55:02 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 18D22A0BEF for ; Mon, 9 Jan 2023 20:55:02 +0000 (UTC) X-FDA: 80336465244.27.83BC828 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf14.hostedemail.com (Postfix) with ESMTP id 833B6100011 for ; Mon, 9 Jan 2023 20:55:00 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=i8mCfCut; spf=pass (imf14.hostedemail.com: domain of 3I3-8YwYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3I3-8YwYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wzUeswsih4PBYsAGoTi9IuotyhbGAzkWIDmV0FovK6Q=; b=as5m6XM4ib7crIuEFQEexuXj1Y1/7MDulOOfYHt4HNdvAjgWHc8URqoHSCw5/nkhebI1kL ZadhaWemUrjlzzcvkB3Lq6SaDqTn/PunAKUrJrU1hdXsjBh+uZpAPjNaatfGWTjpaNQkfg ca7wiQdVFacD5eOvRs7QsNiHCemNxrg= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=i8mCfCut; spf=pass (imf14.hostedemail.com: domain of 3I3-8YwYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3I3-8YwYKCDgmolYhVaiiafY.Wigfchor-ggepUWe.ila@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297700; a=rsa-sha256; cv=none; b=SNXtmxDABEKyjzDxJqfycy5kkUCD31yokCwBfScLhZ+UzG7ZV7E/QUr442gTwbI3exw8u2 hRiUijzfKfzUstQ+TXfPjB8FSps87z3O/rnd9EzRZ2tg5ZKsv96LZo/rQRtm0Q1ByBqWNh IFn1XEqlBngD8mDnP5tL+cPGQ3H4PA0= Received: by mail-pl1-f202.google.com with SMTP id c17-20020a170902d49100b00192be705f76so7049859plg.13 for ; Mon, 09 Jan 2023 12:55:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wzUeswsih4PBYsAGoTi9IuotyhbGAzkWIDmV0FovK6Q=; b=i8mCfCut8WK6+zjWPXPrBAn0OVK3TZZqUir2pgpkQNWhenu43X3/akUsts/8WrGL5U Ag1lqNnPrK3Lht8c6bYeEHv2p/KYkV84L0N5RKup3Lqh7FNgEiwz2qb9iAaK8aobnie5 SWRp3Ej3gvy6Gx0qa+UPT0Wm9SDIcbH9qbkEiBwV9a/y0P6JGxAF3x2bjosqkuWs4x2Q 7wdZhW31F8FFO8VMgBTx/6RIzfF9caT2DmPvdENgd9fKarg0jWwQoD/+83rrhQbpzEQp l4T3tmRoZNr6BoK1+IwDWlh9AP60sfKufgosjp9Mre5rIBM3jgdbCDpmS0FgJglfxzEq wvVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wzUeswsih4PBYsAGoTi9IuotyhbGAzkWIDmV0FovK6Q=; b=OCo5vGkfTO1CK1pXSHH9PEWL+oQIcs/x5kj0QCCMjCxs1jN1mGOLRdtUzXS9se/miq +sfeKrWfDQVbfEw1ghNtCoF29k855KLrLPNvrEbO9k4p/9VyKFfhIGxB1Sj1BPV7ZP5/ snzpq+Idp0Nu27oe9PyGAabcm0f2Wy1sfa2VyHExAgYAmYdYxqgOTDsZKQeIEmt48BOR ymFJoBd3xetp/KosYnmJJCkKcyF3JquTDNQiN6umW3szcIXv5pW38786ptNWLAdIRKMo hasvCbm1vgC+4/FNU5jENS2LaZohRNDegpPKu+bBFnYrglNTDbnPh7KhNX82oJRML1A7 yJNA== X-Gm-Message-State: AFqh2ko3/zQL1deSMjyO+/5ISlb46kD7lGodL/5vFDwVBAwuNWekGP/p B/DSGcZAsuU8uUu6D97GDmNowTHbZLk= X-Google-Smtp-Source: AMrXdXtXcDh9ACK+yDPMFgwqnCidD6d48HBoOS8XbTQx42HB6Mm5GZSyonzBvXPdXaN+wlg8/pxMUTj5geU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:902:7d8a:b0:189:754b:9d9c with SMTP id a10-20020a1709027d8a00b00189754b9d9cmr4530740plm.120.1673297699405; Mon, 09 Jan 2023 12:54:59 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:26 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-32-surenb@google.com> Subject: [PATCH 31/41] mm: prevent do_swap_page from handling page faults under VMA lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: 833B6100011 X-Stat-Signature: sm48e9dp1og69fjwexsk98mr11kn35x7 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673297700-820255 X-HE-Meta: U2FsdGVkX18v/LSUtN8X/lEJfawwe+dbKuczPLPqCLxxMvphtLBII0YkA+Za4HQ45V/cbkcTo4kSIZb+TKzV/Rp/bwrDzD2orC003xWkCEBMzubxwZ9M9KjQ63Xz6kumA9sP3CL/57eFbaQXM0Qa3OOpk4YbC7n9g8AGnJvBrXfxRu1Fl+ZCM5PDuzGbenDEkk8YOEtG0wsLe05BPgSo02HVWtoNNvMJ39CH2nHA+n0425UPvv1s34tUUWttJ2wsli+hpQKod0wUW7+RqwTnQjEbH5n1MUmSx0rgQgbrO/rV+cLI7l5hs1u5fXBusoagliDqdubZaCSK3m9Tju/4YNNlvrr4w8pXUejHoKNRqUfi4E8dEMYB8Ifg8HpZQmMaDLeyV/152EQ/fxdQ9YdRSAEyRQ2E0eIuZ63A/0P90/sYpj9jQ0pNoz3X1TTwWvMxSEKRsbrmOJvMHhwd7qP02dknJ3ef9TEsJ/mwDTX5ZKNP8Ln69QWsyK42F9iccaE4pPu+s8X5vR8mF9/IHwJfRC520GTAPWl9wzWvTHOwjOln+89i3c78Ayxh23Q/+c7vh+cbx9PAQ/pcokln2OHhuzWUxXsIXvkswp7/eTovM614f9Ln9xqiKPP8m9BM+L4C8KQZSHmmmzvxmxxVeQFDN9xvLyTF9O3xV0j8+NFYxZMek1YuerfVvWnbNjMlGsVsKrGeT1tWI6O6TKuXKSazgiSovECTQrpeX/hNYw2siDDV+6881JCrg1G1iXUk8dFzxmpUCHop6gihv/WvnMPaqoZQgsXKamrGfACUA5A5OR/Rdk7uQ73w/sEmmsuWp55+Cc9dpQoMlv/OPdjslKpZI86hsHJWnq3S/SHNGq54aBdH4ih/t/QOgkHQbSpj7Zuq2dcmtCOgO77VwT4n6IrooHAmPoQC24UMh2c9zhhRJ3JVIAWsxLX1nh2B9w4cruc9I1Yqzxid7gSt1c+NjB/ gBDPkKVW 0NS4nJl5ZXWhBaaPPDOkFXXcRL0x51wQgPiiD72o5ejRB1ZqQWyTDWEbWmanG7oIjMYaJyXqb/AjntdsMi0N196SbaDJzEWAfiatXoYfHKOZ9dcdHlgTG2GkdvTi0X7E6gch2HfsrRSuEmaRkyKnVrEAecaEcpx8+6CO9073mOVecEgBeDaj4WsGDZ8h6itRvKhcwAUvrkAtQSvdradYeZq/HfdUkyd45D8ERjsQCZDFIe2CXZebeqMgVXrpvPou4u4JCUZgbgtehaKXKL2jJQpgu5UwrVUrNmKuh0zQh9TsajTBTMXwcUf+FO5ImaErZu+5OTwA4TSKcLxrpq6C6vjvkzsc0uWTZlTFFtBCELY+MrBdSMpxJWzpK3xfS5gwNTeDQvEhea3r33AWhI7N0ZFpGjiIi4icmAGaR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000022, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of do_swap_page dropping mmap_lock, abort fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Reviewed-by: Laurent Dufour --- mm/memory.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 2560524ad7f4..20806bc8b4eb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3707,6 +3707,11 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) if (!pte_unmap_same(vmf)) goto out; + if (vmf->flags & FAULT_FLAG_VMA_LOCK) { + ret = VM_FAULT_RETRY; + goto out; + } + entry = pte_to_swp_entry(vmf->orig_pte); if (unlikely(non_swap_entry(entry))) { if (is_migration_entry(entry)) { From patchwork Mon Jan 9 20:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094298 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B25A5C6379F for ; Mon, 9 Jan 2023 20:55:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DC868E0014; Mon, 9 Jan 2023 15:55:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 466678E0001; Mon, 9 Jan 2023 15:55:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 306378E0014; Mon, 9 Jan 2023 15:55:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1FADA8E0001 for ; Mon, 9 Jan 2023 15:55:04 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EFEBA160F27 for ; Mon, 9 Jan 2023 20:55:03 +0000 (UTC) X-FDA: 80336465286.13.7A6CC64 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id 49FFA160006 for ; Mon, 9 Jan 2023 20:55:02 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=b707Zl9+; spf=pass (imf08.hostedemail.com: domain of 3JX-8YwYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3JX-8YwYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297702; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kEbjJObwF3X8F3sFkf3mioj7FiTBVSLxL3AOZVg1QSY=; b=flg5we69qD3hoUMbeM/Ea1UME9LGRf5iOcJsBJ3XK9G0+/5mK+jx56wyDvPbMfVca9+4K7 vhlAn0X8655W2u7QrQP9D4uzbfsndXjztPljFmjjdOv0NQ5STJLpDWAc89Nf0AAH8pmtFd H7liribKK1+qJg7wTeawWZYxB0hnGZI= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=b707Zl9+; spf=pass (imf08.hostedemail.com: domain of 3JX-8YwYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3JX-8YwYKCDooqnajXckkcha.Ykihejqt-iigrWYg.knc@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297702; a=rsa-sha256; cv=none; b=CfDLpFV5IeFJ1kShF/ieiF2pI2rFZdxQVIg1KIt0rTsnvAAC37Eum2oz71Sul4PiuROgJH OcX+5SZyHz+B2Hla3EDXuDF+RqydvHndMJ/BnaUEP2jMxsT8L6SQOCbP1GnAJEIpmMVL/T pI6jZsSr6eU3CngAfLeqoeotJMw7V30= Received: by mail-yb1-f201.google.com with SMTP id z17-20020a256651000000b007907852ca4dso10293520ybm.16 for ; Mon, 09 Jan 2023 12:55:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kEbjJObwF3X8F3sFkf3mioj7FiTBVSLxL3AOZVg1QSY=; b=b707Zl9+U9/mZ9c+XkHaueCkegKOCNYI/oVAEusRFU63I3srOxecFlDkxyKx//4mW5 JwP5obEgccip2H4R9+fiuFL4CW2k5fc/GSQhSK1HsVxo3ZKjTR0Lux66eUg0E3PQlBTw wXh6GkCABKuopnTH7q3BfoKfyBQ/HE6fsvI1JacGKFSqz7wW0ul/c07mqYyM4k4N+tIJ sPYdgU87tyvftSSm83lgw37DswrvUBJMsShqXbiuQjXDGVNwSgoByZHvozZ1Xmfis7g3 cB964lLozfT6WEMDCdFsl0vR2EiUHB+3pHP9fkmg+z92l6SffusdjR7B04cQIU05PEZE 4Ang== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kEbjJObwF3X8F3sFkf3mioj7FiTBVSLxL3AOZVg1QSY=; b=z82BNLrpRDtCjZR/QaQou3gqEkzY9pw8RjNdMPsr+9sVvV5e1moDNhATxARkxj36Zr kpAzn31F6CgPTs3InmV9PoChdOgI7DKjEg1aUwh02A7y0tuIEjry12bZvynRujagOSx0 erm9ko+QknSOan6jLE0GgK3IeY5bepljp14l3unDFAZyUaVaW3WjafDGOk4+Qb56nBOq qd9krtiHl3ObVRM8yx9TSnISfaxwrQaOEBBOr3sl66edphccxqtLH2K/MF9xNIldwCAU 4mHlxYV9tkRFaaQ2JkEc509ZxVYGzuMsyz/FhxHfujBurkQFv6VTuWaOk3qtDwvsoW37 9Q9Q== X-Gm-Message-State: AFqh2kr6B2FRD7HRiYQUK3JfUXz3Ycih0wj8BtHECFAs4TFtVkYuBw7e R4s7yDMgEvshFZWS1u7Hfj140nr51iI= X-Google-Smtp-Source: AMrXdXuMlfQ+hsH5bYnRbFVRQKfHDnoMxNv1bo1mLunTWqhQUT5+3qxDxTtfm+RaU7eYfnfY1D8yBkPOjiE= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:9304:0:b0:7bf:ffcb:79d7 with SMTP id f4-20020a259304000000b007bfffcb79d7mr163411ybo.446.1673297701437; Mon, 09 Jan 2023 12:55:01 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:27 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-33-surenb@google.com> Subject: [PATCH 32/41] mm: prevent userfaults to be handled under per-vma lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 49FFA160006 X-Stat-Signature: sgk348qd4hk7hiumghpsqbiedjxx73we X-HE-Tag: 1673297702-173652 X-HE-Meta: U2FsdGVkX1+Z/oEQivW1Xgp000VKhMSZ8qZrvGbaoEZMLUDIV7FTQ1Ql2Ccj5s71v/urn5NX9YTQWernUIj22v8A5tFDSCS8x5hl0dGS/OFJj+Ir46n0/u79ym5dk6wCDaneL6n9pFRd7O0yXEddUQP52AxHUVEMpAPj35Ed9vFXjU7KJFz39YtliuuM8lUTSJwRCOZaDNGYSbtVl6ehCCTezhEKVUl7Im+Y7bH5f1TeaGsKtTTFK145CeZrW98kCsv0KoVgWT2+YHmKPQf/ThxcD6hQAcfvl0RvYR4qSD3FLMrgfwNE1SPc6I+qPZnm8Pe9ljDqyHGfoscQuDYD9ni1Cd4rH3mli+oh6jMXzWoBlcKinWwWTlrP3qfodFBP5QLxnkSYG8WPrSk4hl02Rq/bbWhteaqlCJ7w7uqNq4rNZisqfolYDH5yXzvHUdEyKex4sL9S0nV9CYA2eX2cscVcURRozCFv4X32yXQb497tyGRcv6iv7rmNmiANxpKFR9SG0qrK0tB1CBHcv1B3+Ta9JFLQcrd1GMlTUpmoAhUzGy1C34c8a+b6pge6YeY55xLZBBdUYj9e0TwERer1PDiIebWnalncnt/FQ5DfaNjxOz2bWkR8h/TyZMXZ1B7IHHOTXfgrtVYfXOaG806AK1grT/E7ZZhUFDXiaLjb+WeyOIcJuvAEvcrL976P2YcWYsA7p1Rq/YK8y14yR3tV9lpBR8LJQJh3M/ISUhRArqD4KL05KL81QdQpXd56kjPDqqneMPsQziLDvQUmojYXm7oDuiOjNED43vBZ2NgHXSkHdpYUSCScHHc7sFl250onohta/Jeu9DuIZfvyBgTmsNHK9pU3D+NRo4oiEFzLiqMR/TGeTA4skOeu7Mr4V5iWtWDF7O0wvwaO9241VBnnxz8ZW49EftchEetrTAAHDpYesNbWMQ1MRNmFCtxJE6vhqvukyHbs9ItBFt7Ko+c n2L5p3iS 7rrocRifvlSqPpmPX2Pgb945gcnIlp1kq0Cd2sYhmO0tSVSEp3Sa8kZMFkNqPNM7Uq6SxOrarHxbPSv7E+NMXNsPbxcnVbVgx+NJsagX2FQbkdYQpTOOygmYr/xqcGdsAK7sa3kUqX43Y3x/7g/ldwYbi/SKlTnxTR6eMkzACW0UwYUMTGC7wlVvodtDAO4tuUBBc1IXzP5pb/+cfmqTkciRuOCZr0faiCRVeSQKaDbkjSCd68s+gpULDneSmJQivp0H1yXesBfR6D/wGqmHaNn6k5A63Q7cI49M4oCU6GataiAt8GmiciGLwEbrWvYoq4wg9VzJ9uRNltg2cIDtHoTQx5sbyWw+t8/M7oJ3IHPAneSc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000060, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Due to the possibility of handle_userfault dropping mmap_lock, avoid fault handling under VMA lock and retry holding mmap_lock. This can be handled more gracefully in the future. Signed-off-by: Suren Baghdasaryan Suggested-by: Peter Xu --- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 20806bc8b4eb..12508f4d845a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5273,6 +5273,13 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, if (!vma->anon_vma) goto inval; + /* + * Due to the possibility of userfault handler dropping mmap_lock, avoid + * it for now and fall back to page fault handling under mmap_lock. + */ + if (userfaultfd_armed(vma)) + goto inval; + if (!vma_read_trylock(vma)) goto inval; From patchwork Mon Jan 9 20:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0A21C54EBD for ; Mon, 9 Jan 2023 20:55:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6330B8E0001; Mon, 9 Jan 2023 15:55:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C3B4900002; Mon, 9 Jan 2023 15:55:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2909B8E0001; Mon, 9 Jan 2023 15:55:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 108648E0011 for ; Mon, 9 Jan 2023 15:55:06 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D21E7160F35 for ; Mon, 9 Jan 2023 20:55:05 +0000 (UTC) X-FDA: 80336465370.02.F0611FC Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf08.hostedemail.com (Postfix) with ESMTP id 4D21516001A for ; Mon, 9 Jan 2023 20:55:04 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=I5umo4kG; spf=pass (imf08.hostedemail.com: domain of 3J3-8YwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3J3-8YwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297704; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mbt4mHoq0eWZO149X3WH8AIvdwQHHBNP+0MBWdKikso=; b=4h3rW3ZlUnxTurVCx8Vb/nwWMb4rEc0yeNGnpUBJTQcQXWamwlAk4/J1aynyNhw6MLf4gu N06mD/Ted6IlfejMEal2xB9r4FCgf7xOvSY4S2EKBAZO13qeAZANr7nCRBjlNNwnKL3XlU 5k1xXcDmMR5KI20JNp4LiieNrQM9noY= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=I5umo4kG; spf=pass (imf08.hostedemail.com: domain of 3J3-8YwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3J3-8YwYKCDwqspclZemmejc.amkjglsv-kkitYai.mpe@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297704; a=rsa-sha256; cv=none; b=2c4xnAQk6P6UT/NRukXQbNtjRvda4U0Qckec2AKST0DEF7khlmPvFkO6HLuUv3McAgMsQ1 ALifjO0vccvBMsrmTH6DWwuAFjKV0GQFfr2LYceUn6kIm0IyiBgr+nc7a5abtdEvF6vLqt 8THKo6DnOMtn4gFqfPhsCIjueiFvr5I= Received: by mail-yb1-f201.google.com with SMTP id z17-20020a256651000000b007907852ca4dso10293625ybm.16 for ; Mon, 09 Jan 2023 12:55:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mbt4mHoq0eWZO149X3WH8AIvdwQHHBNP+0MBWdKikso=; b=I5umo4kG0hMiq4gs5OujcHd7Sn1GGJbiybdYQYTj0HAR3QOkD0OA35qcaeXUMRzGAJ dIG3EBC52wC5ibcKgTHA/19I2NomM7CaXPc78eVykHWlIF2Vt/KiNizKg3yGh0eNsGHP jtagKS1D/qcsRmP7Fqc869E/kUJC8aHONABQKFab5CUW1r0+y/kJjLtE/M8OXtv+HWMx 0jlTEySwEhV749+tPTyOU0MfX8AR34H6BYDnkbNQmdajk1IIrv1WChV6C7Fwj8u25gID LgF2w8/r6WitNZ0Roa2WVZN3Fbrz2yHOP70XPp57APsjpBjV3GkkefNK2oUkx1yrmoLm XvuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mbt4mHoq0eWZO149X3WH8AIvdwQHHBNP+0MBWdKikso=; b=o2rF5DyMisVZL53U1/vZI3h+BGrYq5GPR0Hr6XK0wYqy2D9P2bqYhenvAcvRlF2Tal buZsUD3lu4Sar2ybDkXZgosWIoxM3dbiYhKj+I7YA7oEZV7QXIpZnzcwqJ+Q3GF/wpbV 6F5EUZJZjshXS8X0MVBLvf1dY3N5i2PxKuoZtDCyAftZp9gI/evFEKdVVww+tC1D3si6 gCeZtli7CVjiFBtxplB/8MdV6A9hNMzK+HGdO8GuYr/dl2bvxtWNEgRdlA2PYlk4Deno ZlhoLbcXGcPPS1ydZ5X6NcuvFdIHmwyNdpEVS+Bcxl+jZa1MIfEIB+LnT9/+aQFNJKA5 /Grw== X-Gm-Message-State: AFqh2kpVqYz36FK0j1WHapXnUNGje3xiPLeEzCQDK/o+19ILoBowSRF5 Me6WTRiVnJHdn2UQmttaz4XNPOD7YX4= X-Google-Smtp-Source: AMrXdXtYuIZxYJtCLtoqxn2YJzAdgg4PkVPb1cvNxFA+XZbpVDno2DZhvITBpFn/vgBWgY0+QZ+ehdsSMWQ= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:9d82:0:b0:774:df47:b5f with SMTP id v2-20020a259d82000000b00774df470b5fmr6154650ybp.405.1673297703709; Mon, 09 Jan 2023 12:55:03 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:28 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-34-surenb@google.com> Subject: [PATCH 33/41] mm: introduce per-VMA lock statistics From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 4D21516001A X-Stat-Signature: t6wf7uuu6ua1reu5t3731t3n91tc53r6 X-HE-Tag: 1673297704-82493 X-HE-Meta: U2FsdGVkX18jmrwfoory4xuIAHdV9NNEfUKxAi8xHugZn3JFXZiGLJPQminp6SZ4kwgF6bEiS1ybxhaxbhK7UsoLYuknP2dqaIISaZk5/UqZ9ImHTLH7PL09wgikZhLyyfA7dHx7Ic4yj8nHZ0DGvN2YDHCPZg8MXdPjoARS806jWIfyUTgzy9tLSwavCbriZVMw8PCBgW8YPF24pKVwwc7m6yQLJACq9Iy5fwuzIVKn3JFiEHVOxwdMhuZg4PgU+QAnvzQqtrjRYND7tpvifRaVj5ulbiAViDlbZBvrCSd9hiYDczRYYeGXb3CNzSClTSSd7lGbhDVQ/B8PmQ7hwNEODZ4rEk3fkijUNJVC0qCUzwyx0/fCDrzxCLNVi0iE0jSIS8GGuOwLhZfoPOgW3vCFchvTdQ1SgaPa4iQYVadzV6iUBH+rr+4mkvyMX5WjgnzvxlL9lNTUYe8w9/Zl/5/XR4ZGGiuQD42lP4nV5PR1ivpQQtApN/wVP/ELMqcsIOtdvqTeiZLpLSowm9/7m/29l44tg4k+bHN4qKdDq+08/7hvMYVu3goN+LDlKmXskU+WVXalH8yH0W+/RrIYGAXWZhmgjRUyA0kARokBJaN+BHHTcd9ASwKnDBVTHvvs5Qx2nq0LBb29I6xbJuH3gTgKZAZ+DqvgPz5DW6kcP55dz9hmg7mEyRJngHd4iJOfbnHnJ+6P7gvad/W4RAYf971UVOB+7dhIVuqmhNd8VMh7vwl3j2DL6u1Sn8hZO0JVr2I90QZWcoluXIYQdmetb2YT9rvE8XEzHCUnuPsQPdnX/X0DQa64ub4C0f+1GDVPNzyXeYOedtp8Zt+BlFwc8j/01NRTcX6NwBQu6QZVnK1Jc4NCZh+mzExAFzPuIK3mpK04HlnuRrIZkj12tjBeVn76dmmEkcBSO7j82MsyrhkwhaEhPAU2sbN78UQl+Q7b9SjNkylIQz6Dcxf6T1Z IjqXSXzC cRTMBPYz1nsWc5c32qaCa22LkHOljrlx6iadCz9g11u3n+j4/ePWhNR2IgmItKUkhtrecBsF3c4uz0pVOuDvxN3+bS70YSDkHRn2SMJs7WaZzCsmhc2irUB39hxyxivKUqog2h9AfrwsHq5BH18bDVTdD+hVUCF0862sCv3ZWrx4WLvbBNqi1Ugb+6uQj8Atwv/a3/Z84mA0SOkJ6nJRM4KK69BPvhMQsEuqaWsfXoIK+VdnE21aDKKbLn/u5P4YNjglloLnUncscE1Yyf9FjF2gWyiZ+zfzI75PMrbrfUZgMhbvpv/fpoeopdrVsOInnXpKt79321yk5FIEa6KmAB2Qf3mYcScI2/jfBWy4u/yf7aU0lR1XYhs4Nzg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a new CONFIG_PER_VMA_LOCK_STATS config option to dump extra statistics about handling page fault under VMA lock. Signed-off-by: Suren Baghdasaryan --- include/linux/vm_event_item.h | 6 ++++++ include/linux/vmstat.h | 6 ++++++ mm/Kconfig.debug | 8 ++++++++ mm/vmstat.c | 6 ++++++ 4 files changed, 26 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 7f5d1caf5890..8abfa1240040 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -149,6 +149,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, DIRECT_MAP_LEVEL3_SPLIT, +#endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + VMA_LOCK_SUCCESS, + VMA_LOCK_ABORT, + VMA_LOCK_RETRY, + VMA_LOCK_MISS, #endif NR_VM_EVENT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 19cf5b6892ce..fed855bae6d8 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -125,6 +125,12 @@ static inline void vm_events_fold_cpu(int cpu) #define count_vm_tlb_events(x, y) do { (void)(y); } while (0) #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS +#define count_vm_vma_lock_event(x) count_vm_event(x) +#else +#define count_vm_vma_lock_event(x) do {} while (0) +#endif + #define __count_zid_vm_events(item, zid, delta) \ __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) diff --git a/mm/Kconfig.debug b/mm/Kconfig.debug index fca699ad1fb0..32a93b064590 100644 --- a/mm/Kconfig.debug +++ b/mm/Kconfig.debug @@ -207,3 +207,11 @@ config PTDUMP_DEBUGFS kernel. If in doubt, say N. + + +config PER_VMA_LOCK_STATS + bool "Statistics for per-vma locks" + depends on PER_VMA_LOCK + default y + help + Statistics for per-vma locks. diff --git a/mm/vmstat.c b/mm/vmstat.c index 1ea6a5ce1c41..4f1089a1860e 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1399,6 +1399,12 @@ const char * const vmstat_text[] = { "direct_map_level2_splits", "direct_map_level3_splits", #endif +#ifdef CONFIG_PER_VMA_LOCK_STATS + "vma_lock_success", + "vma_lock_abort", + "vma_lock_retry", + "vma_lock_miss", +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Mon Jan 9 20:53:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094300 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7349C61DB3 for ; Mon, 9 Jan 2023 20:55:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DC5B8E0006; Mon, 9 Jan 2023 15:55:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5ECD3900002; Mon, 9 Jan 2023 15:55:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4669E8E0011; Mon, 9 Jan 2023 15:55:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2A6BD8E0006 for ; Mon, 9 Jan 2023 15:55:09 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0945E1C6599 for ; Mon, 9 Jan 2023 20:55:09 +0000 (UTC) X-FDA: 80336465538.14.35E3E5D Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) by imf28.hostedemail.com (Postfix) with ESMTP id 67FD0C0008 for ; Mon, 9 Jan 2023 20:55:07 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rw20v86z; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3Kn-8YwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3Kn-8YwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297707; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wQaK3IYB3deWxpAe9eNtEaAaK5GLZai42ZGvF8iLG5s=; b=rbu91RR69zIArHJFxDQvQf22csTtN6612pGhyUsQmQfPAgr2uGBi4IWNKvAl7iag3SIEqF lltDscDoBr3ynMo+56fUiinWYtb98oJOtKn7GydkdGhmK9ZW93gnFMAsBCBg+k7AAh1Sbf 5zuKDNvUSZrvLYgVWobcM0Qmm2Bj6Qk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rw20v86z; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 3Kn-8YwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com designates 209.85.210.201 as permitted sender) smtp.mailfrom=3Kn-8YwYKCD8tvsfochpphmf.dpnmjovy-nnlwbdl.psh@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297707; a=rsa-sha256; cv=none; b=yPNwQ7WjINU6JOnXBckGBNDmk3QQqsHgUg27gQdtJql3UywSEs1fqBCr8Kmj3KvFcHm90r glrStYlawMKdsMunaw6Yp415eeI6McLX2+ssdvaMJ1j/TnUHPImU/HCUVwupaIaPCmIMPi bNpJMQFKFWOyGEqXdahhdKZ9BU14Pjo= Received: by mail-pf1-f201.google.com with SMTP id m5-20020a056a00164500b005825b8d2723so4019846pfc.21 for ; Mon, 09 Jan 2023 12:55:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=wQaK3IYB3deWxpAe9eNtEaAaK5GLZai42ZGvF8iLG5s=; b=rw20v86zZuVQYP+Ot38AI7CSNxJo3kS0fh4+Xzkg0ua9O1bePCbmIGXyRAOceAwONa J0TZdcxQSasqUL/oBe6wf7jLdD0R+p+Bu2MPqV1YykB1eqTKi0/qjL5rdUZIcmvdelEF RBoPMFMogBGK0PrycWwKvVRHJ+Ch4XmPWscy4TJxhfZ7irUJVjF6OWFk+4uiaZ5g0a9h xVEF8daEuEJDm1wmFyKFMZlgoKpmxoJhKWB0BV89iFmA1f0B2X0D/WawWhEeTcmbbuya u6sy1rh3eP2M76YM3K6bmEc6ANBDahY+/H9HaeO8ptCkobUJsiIXuerRnexRjBXA4ux5 pQew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wQaK3IYB3deWxpAe9eNtEaAaK5GLZai42ZGvF8iLG5s=; b=TYZAHMNbsdiIp+8b2YXIF7o1nSdQyYJqdWxdLiSQP9KMCd6F18UdFmyjU55oOucBYo iWAJb3wQ8SVKvAeiC9aZ5MAEmXE5APwPXougNDtP8FIrkWqo1GB0BrBb2qHIBlhjOo// 0SmcLWHsbzjFVbs3A8/swiTELaFCmn09tEJuFvoz7sgjySMXFcrzL6TTASirRMM6Pxkl 8NmnPEr4usA1fXAiCQVDLQ5bcZ871ZT5NyupqfWWIes2/ppYiDep0pYBUPtQccsLZuKS 22pwubFEPNXlHDe8ALjPJ2kIcHiUT9qU6fvWb6KfTtQugra45TnQayHIdlwBBZDlNkb7 lZqQ== X-Gm-Message-State: AFqh2kosOjSaQ7SyrSOh8FyHjA57jL5V0yDxFi6bOnFVu9uWJw+EXsZK kEFBJowI6R5lJ2AA6TdWau3eOv8IIzg= X-Google-Smtp-Source: AMrXdXslCGRX6fAYSmsr9gs68MBt+wxNt6fnptg7xXYf9+oinmXIkRDOo+t8kmB5UcEQZNFBtsxG/pDk5Sg= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:902:7583:b0:189:bee:65ee with SMTP id j3-20020a170902758300b001890bee65eemr3889654pll.107.1673297706216; Mon, 09 Jan 2023 12:55:06 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:29 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-35-surenb@google.com> Subject: [PATCH 34/41] x86/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 67FD0C0008 X-Stat-Signature: 4bfk1twb4o765adwaxe4o5b5bh6fmx69 X-HE-Tag: 1673297707-956473 X-HE-Meta: U2FsdGVkX1+AiDt/8cdz2DwdkOQWcrXyeiUO+VzKudoUz/9YJ3eU4DcTd+rsO0Cuap8cz5QLGEiL2gh26OuEeFycA7tQllD3AsbziPoKKAIhGGqxRVoqlppyXEZaB4vsK0OGQhhE8F3Wc8cjJ6aPXHL9ZMdCYs+AjVjBw62bucTG5Wf/0QmOURZc5Mn9RXea79v/rleAE70tZcB37KqyaT/dlmIroK0Vk16QTYYg60ZOcQxgyvh/DlB6jEPOZ2lugqMDYNknPYI2JDYM3qSurQyVvRq2POyf9sfimJxCWhCavelsw0MwxLxl8tBM2Cgc2fr2HTVI2dQu9kUygB8MNCfMRz3PhIVbucWer2M/XeNjKJkz12cLkn2i7ZSsjnSSWmA9t6eVkvM4Bctdy1yNWGmkOpRaxkKf7+VGsoDLqzAm1BUNKrecI8tg75USMvB6WJTFffLHEmEPib0uwpY4ysO1URsos85CCIata/3WFWr0GEtXJK4hdGd2Y4xCgsDHv6nzHnKKKnwqQWlKDXiF6x4xyf9nSH7zFUJL+OpuBwJLiAvQtBGK7czbcWO+p+nBBsVVGLnH9uJmMX+apfWpVMXbzrcTPRYjpphggidfWJL/geRITWppSiFMTbPmU78ZKyzOEOkAZrx3meZBcX/RpbUr4YU1whKjbZLK+fjJzhzRRucwptQi0chIoCUzsIO8LY+l2EdSlIkzfq8WLfhU77adj17yzS+9BzogqTXAdiVl1gYSIy2R2apedm4MiTzejli4EEvriMe+6YD41lb5gsFV5YGU0hyHsiT0RU59J8EpuE4fZ7WJt7527JBctOYzh4B7/ivv2r3Y2+nXeNrUsBWBPIEMU2NEicDIKoSAm2/BubCuGpYonJg+LZ1Ixy+AGv7TsvvV6j3OT4I7l5j/smf4Y8MoJr0YkD2Qu3XKsF63sU6UULeAtLpPDT7ht6C46KyjIUt/OlLj3it4Gx0 zg8SQ9+K ZG2mdxasZaSpcIkHHPzyy6BmuTtcYMox73OUf02/SbwQmVZAF2wejKg6dw1vZJFc8BfyN6p5xqa6O0XRIXRmNhUyEbh7YxSk9mPaG10Khy5D7HivSJlh3xIVCkIit2O9rNtPXaMG9/iPEZB5N/YS1rDRq7nTazla6bzxE1dfQSOtychewKi59BTg0DrgJMHtz08jw2GrmCngUPlN+wtWV43plHsR559/Nf3JOda+TTuBlZjL6GHJzu0TvHaqDUAt6UVvb5AU9Xbcjz4hmhhAywYDCDz5pT8eCklCrRJHXeiNdiTGl+MWT/fm1DccZFn5efbnvZjqZtCD0SHl+bDN1E99d7hlat1fy40VEjSS6JnXJfW9Xt6se5qSvag== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/x86/Kconfig | 1 + arch/x86/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 3604074a878b..3647f7bdb110 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -27,6 +27,7 @@ config X86_64 # Options that are inherently 64-bit kernel only: select ARCH_HAS_GIGANTIC_PAGE select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_USE_CMPXCHG_LOCKREF select HAVE_ARCH_SOFT_DIRTY select MODULES_USE_ELF_RELA diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 7b0d4ab894c8..983266e7c49b 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -19,6 +19,7 @@ #include /* faulthandler_disabled() */ #include /* efi_crash_gracefully_on_page_fault()*/ #include +#include /* find_and_lock_vma() */ #include /* boot_cpu_has, ... */ #include /* dotraplinkage, ... */ @@ -1354,6 +1355,38 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_error(error_code, vma))) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + kernelmode_fixup_or_oops(regs, error_code, address, + SIGBUS, BUS_ADRERR, + ARCH_DEFAULT_PKEY); + return; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* * Kernel-mode access to the user address space should only occur * on well-defined single instructions listed in the exception @@ -1454,6 +1487,9 @@ void do_user_addr_fault(struct pt_regs *regs, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (likely(!(fault & VM_FAULT_ERROR))) return; From patchwork Mon Jan 9 20:53:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA5A8C54EBD for ; Mon, 9 Jan 2023 20:55:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8AD4B900003; Mon, 9 Jan 2023 15:55:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 85EF3900002; Mon, 9 Jan 2023 15:55:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 724DE900003; Mon, 9 Jan 2023 15:55:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 5578F900002 for ; Mon, 9 Jan 2023 15:55:11 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3619540F2F for ; Mon, 9 Jan 2023 20:55:11 +0000 (UTC) X-FDA: 80336465622.27.0690E49 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf16.hostedemail.com (Postfix) with ESMTP id 8A1ED180014 for ; Mon, 9 Jan 2023 20:55:09 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="AELPgr/6"; spf=pass (imf16.hostedemail.com: domain of 3LH-8YwYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3LH-8YwYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297709; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=m2UwVHuB7txMBXnAywM8cXmBB/zrDJgCqgF9heuZ7RI=; b=zMUQWRkIB9iBd7+oWnebMtzOhkRrBS59zT6n+BXkIi1Xx5KY2jG9ylWLN5oIHtQQdlO4H6 dh64fCmgaqhBiBE8EsBI9ebEGTLiddi+/dVZ3JU83QqobPCAWI5CFwr0WaBWUybY5QwVPF /vndk/c2/9GR4B1bU7HYvRdgMQjwQDI= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="AELPgr/6"; spf=pass (imf16.hostedemail.com: domain of 3LH-8YwYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3LH-8YwYKCEEvxuhqejrrjoh.frpolqx0-ppnydfn.ruj@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297709; a=rsa-sha256; cv=none; b=ZavCvG6d+DUs5XRc/CM2ZFel8ghIn1MLcFWnai3HAP2HNhRJAfTbOwwgqt/tp1EDU1ul4E O8krilGX6vOV5hgzt/NaUC8jvZRRPxS+XuOx4/sk2OF94n11+w35uPxiLBJudTIUHo8l3Y FOhDTvhJRYmj4ucytxhu67hrvBeR0iM= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4b34cf67fb6so103931247b3.6 for ; Mon, 09 Jan 2023 12:55:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=m2UwVHuB7txMBXnAywM8cXmBB/zrDJgCqgF9heuZ7RI=; b=AELPgr/6hvWzT+1V7lD4mX315E9t1xeapwUzcEqCdKIYPiCzKk8dc+T92lAP4lgkah 7yRjN7psrT8Aldeuq4taS71dz7N2REjLgY5gl/EmOw6Bjz3k1TLxoTZM4lDuQnOU9Jya +UewXZCmP8ol69UqHnariEjrKTpgLVgEy+/Bgrxiodf6VRTx+vPl+chOix68VXTazEey Yz24CBUMjgzvQiO/GtdPs+W2QXpN/PA/WPYudNeoUrMmtRKn07dYT94LClSxQvVwRpgt N9P9H+WlrUeFi7Q8PmpAS+IgzxksFHRinwxPnTAAPwB27jZBUc2AgIvIZ8+W58HDzxG+ RgCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=m2UwVHuB7txMBXnAywM8cXmBB/zrDJgCqgF9heuZ7RI=; b=jPNiWreoC4tEZ9G5c1U6k1ItGFIQK/gJi7z+OXoUPYGkbYSVS3vhIJdD9bLw2MTaBk dBXoy4WN5r8tpsx9RLDoiSjyESYMWIBCODSh+axfNY7bp4FM0xRG9l23Q3QACV0ukn72 fFNXtZVC9k6PLtIxctzQm9YEiq+KwUY4wgpSAFz9KDFqYWfkB4P6d1RAEKxX9C8gBYGI 7c74Hjw8Aq/7mhAu+YMRvIPPzHDweT2PX2hFi2p1p5ExBfw/Xj52M5r6yeFg16+2WVo9 BmwFkRnB4HhgtT5T0I5Nz0ZvZ+6ZAoGLmPPwlBdT5CCPSYdDGF4LgiL+JEdggsAywrOX d1gQ== X-Gm-Message-State: AFqh2kpHJxaQs2q35boqOdX1p9SD87aFr9tMGQ0/9zMzHQCRRxvBImN6 h0Q+ZWY22C/WMSPQmXETnKdlkGB/TvE= X-Google-Smtp-Source: AMrXdXu86J9QPZ8qYzzFeKpvTLXFdkNuTVtCgOtoNpSPm2yn8vVjTEAXybleIDx2Sx/3r3Ulgd8R54EZk2M= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:b78c:0:b0:769:74cd:9c63 with SMTP id n12-20020a25b78c000000b0076974cd9c63mr6699949ybh.257.1673297708715; Mon, 09 Jan 2023 12:55:08 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:30 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-36-surenb@google.com> Subject: [PATCH 35/41] arm64/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8A1ED180014 X-Rspam-User: X-Stat-Signature: x67h5hdbtmyuxaugyzhju66of9az6p4c X-HE-Tag: 1673297709-110498 X-HE-Meta: U2FsdGVkX19tZDXL9tR6DfrfEFvPEydqTSKcPIINq7ghs9rEBU9H3v7LwIGkuVcfA+CGxNNOvs0DvOOd2PzdudyvEhbBf2Ld8nBoJPlSv7VwS+1QRUf2XGDsP2uTHwNVm5IhitofdGuy3WXt7JiY9jd4+0FIiXoSFGxiDTddToU3dEsjy447WDlZLhsy2SWSca6OqgnkErPNbwgf4beGcLR5bTzkmgI1tlY0BZux8GpEaWqGfzU0CLHtc8EWrRiRvu6xLSUaaksly74qHN5F1Ym87XaMAfWKGYPQcKRko7HejQ0kb9UuHehAO0GVifXPgFoe0IiVOdWPu9xeHCO7JWQMdvxbiaxckEHWfeOEO3QPLtp3MqriSQoZAerhdZ/kNk4Zx+qTWaxVPNUe4RxZA3ioiqSryn3ehnCdYq/hdqWvTzFzvj9C9CkG0EvTFAFay4NARQAW2Tzf79NN2Q2AzXRZm9ksEJuD2LChJlw3O0ue7JjLiLlA++nh7EauFNhTyUHRnXVZIwX2isfQ7T5L6spuGpEn9Wget+eFkmPXs8tH4AuUP1GnFmeMShz9RtTr+gY8DgJpU5D7ZSw0+zmsvSNCrQ9gd2uWN5EgT5JfwqZ6lHathYfwIljr3IleLpDznNNQ7LfAsw3Mooq3kPtx9Lx5R9USzWtvIGzGJH7AW3z5giwgTqEj3LyYaD8gDpPFDEGwiJCilbR8+qO0sO/i4VB/YErvNCr7F6McBu9FQjE1EiLRzUnUxe6MDNlzP7rFhqC+TkP6FSzUK4z+lgETxeNONqXOCPdkDnYUS2aVx3/02NKh0OdWjcM0lO4KHeCb6w2I6U6FfEkpmEltpHrv8A1sfgB9ru/o9H5mR8CfiRFYVdZCNV5vgeY1OMk2gdqJJQfXVajIB/jIRE2tmbcB8jFs5v1KG8GGAPyL1IHhKeSHuzJRIiqZJzYeLFEmCTCpKS79tAUbAqTY9ULPY14 2woRi7cz ekv6hs44IJP551BfXqIIe5LB/QHqZRvW7NE99kzG6Extgg+/hXK7qRpdeBBL8c5xTfoSvWCz5ToH2vGPSjOGCTgAWPV7YBkRDWMBINTd70p/icTYmazC9anuiTNFH8kda5ntoFBzjkWJ47oa7aygpTEh5JJRHPWMSIROZ3dME83uwEZf9YmGTloA3oggYIrsDwIWfib5HC/WecvGb9bMbKRmrE14tOE0qJrGg0Mx66NeSMGq0r1CCW0uT7+SyH4hWcShbZDZFRtKo6nGAFNCWK/5osA8EtgEC837l03514rVtO0hfmHZ05gWLJJpJYXqqIT89YZlsaVZ18VzEynfNb9cuYuqhwf2xxGhm9fPmnnzQ1Bk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Signed-off-by: Suren Baghdasaryan --- arch/arm64/Kconfig | 1 + arch/arm64/mm/fault.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 03934808b2ed..829fa6d14a36 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -95,6 +95,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_SUPPORTS_PER_VMA_LOCK select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 596f46dabe4e..833fa8bab291 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -535,6 +535,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, unsigned long vm_flags; unsigned int mm_flags = FAULT_FLAG_DEFAULT; unsigned long addr = untagged_addr(far); +#ifdef CONFIG_PER_VMA_LOCK + struct vm_area_struct *vma; +#endif if (kprobe_page_fault(regs, esr)) return 0; @@ -585,6 +588,36 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr); +#ifdef CONFIG_PER_VMA_LOCK + if (!(mm_flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, addr); + if (!vma) + goto lock_mmap; + + if (!(vma->vm_flags & vm_flags)) { + vma_read_unlock(vma); + goto lock_mmap; + } + fault = handle_mm_fault(vma, addr & PAGE_MASK, + mm_flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + /* Quick path to respond to signals */ + if (fault_signal_pending(fault, regs)) { + if (!user_mode(regs)) + goto no_context; + return 0; + } +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ /* * As per x86, we may deadlock here. However, since the kernel only * validly references user space from well defined areas of the code, @@ -628,6 +661,9 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr, } mmap_read_unlock(mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif /* * Handle the "normal" (no error) case first. */ From patchwork Mon Jan 9 20:53:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01E7DC61DB3 for ; Mon, 9 Jan 2023 20:55:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9492B940008; Mon, 9 Jan 2023 15:55:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9217D940007; Mon, 9 Jan 2023 15:55:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79A7D940008; Mon, 9 Jan 2023 15:55:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 67355940007 for ; Mon, 9 Jan 2023 15:55:13 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4B80A160F2C for ; Mon, 9 Jan 2023 20:55:13 +0000 (UTC) X-FDA: 80336465706.13.EB843B1 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf18.hostedemail.com (Postfix) with ESMTP id AD4481C0010 for ; Mon, 9 Jan 2023 20:55:11 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Lu1lZFQR; spf=pass (imf18.hostedemail.com: domain of 3Ln-8YwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Ln-8YwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297711; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n9fXm3izjyyPtsvowAse/sqFmR2VD35eu3/dkFbABUU=; b=CGOm8JBdmhLZ7C0dwsl9FH/VGUcBvfq1po4DeKun3a4N6fGRP93ziT3UgvIcUfKZlyejsG HShSC2AsM7wDQB4X8pEQc0dlXkrzNnqNAwgK3tWwZqHPB4mFUV1KQ6kyDXY3JzPhUgo6DM XA6BkR+fX6FSlPkaQxPXYiJeRX/34Ls= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Lu1lZFQR; spf=pass (imf18.hostedemail.com: domain of 3Ln-8YwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Ln-8YwYKCEMxzwjsglttlqj.htrqnsz2-rrp0fhp.twl@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297711; a=rsa-sha256; cv=none; b=Zwfunkt40d7arWnhztTGg2XSfHNMiQjeVWzrXFRiAMSclIqUL8Xa1UbOdP/yinY2zIwB3h 64CZpt7347vo08qPxVZpg9QdQRRtoIaaAsjiW3d5VuPuQgpmbnuTdivwWJ2ZXO+uxmn1s+ jUwYLN/z1c/AZaKUtwMitycQzNRx6yk= Received: by mail-yb1-f202.google.com with SMTP id h66-20020a252145000000b0071a7340eea9so10193824ybh.6 for ; Mon, 09 Jan 2023 12:55:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=n9fXm3izjyyPtsvowAse/sqFmR2VD35eu3/dkFbABUU=; b=Lu1lZFQRLQ14pZBfGy4NklmBcU6A/6INZGZkbzsgSlo/xf+q4f3Srx8+Mwys+6coe+ Ffmm0USNrJ8AFhCvinJsbBeU/mUqellUur3KPDhj6fquokQjmdfa6asRXc95A2ZwxK0r 65ObusIEfZRdiZ2acFQ8p3bJxbaCwGG5NTSDswdl0j+wpKeIAqm5hRE9lT+/32AEgmrz B2Z5fNXwd3NAztwL7Tcvgx4jFF9uDwxQg3WbD1oe+/QJTBY53gcrTvhoGSAa89iq0kDV 4crl41ZGwNSTBBEwkTNkn6j9GWODsWMSurEgXbqgc2WbNqPAfDPrxauLcSQy2BQ/r3X3 zHyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=n9fXm3izjyyPtsvowAse/sqFmR2VD35eu3/dkFbABUU=; b=FOGD7jH1pnEGW+8gdM0y/pEAjDZznkSztQ/pat5tmLmSzDyj9qVN002P0Pr1yFKo8J /UJLKussaAMO9lZjOK+GMGq1NyrGvvfvxm+nmOqi2gJF/Va+qZd9tYMxtJ+gq8h9peHY LfLxXBSM6/kxlnT6ng5OAcIA4csNB3xEcdYLS1glhI1NPt/goNOLK2FCdHp+VHWiUuX2 J3qsTo82rRzWL3wFiP97p8E5M/OsJhdI/tbCJbCv3biDVU6vwdkHq6MkUKOYQDctQg+Y yl5Wu2e/Ky92RBPw9tZpNVZir80tB9kjxKsY8zycZ5lNEJFxvFiOCYXk0FoOl3eUhKqf 7eTQ== X-Gm-Message-State: AFqh2krE9izaM2A2XjlwZ5/l5MDkXP5UmVErMAryEDcEEM05oc7oIGS2 dJqvYnA+ahbipB7GRYylQB7VQnNUJXE= X-Google-Smtp-Source: AMrXdXvNqK3Ohvvx3O6oqyebZlpPuhsn/AIt7Mf5iuM9SXcX9ARLOAj6UwU5PecAUl7FhPG0GswhG64wODs= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:730f:0:b0:6fa:3287:8df6 with SMTP id o15-20020a25730f000000b006fa32878df6mr6834426ybc.424.1673297710804; Mon, 09 Jan 2023 12:55:10 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:31 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-37-surenb@google.com> Subject: [PATCH 36/41] powerc/mm: try VMA lock-based page fault handling first From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AD4481C0010 X-Rspam-User: X-Stat-Signature: k7qgfuheokwby67motk3jtyctw5e4z3x X-HE-Tag: 1673297711-33222 X-HE-Meta: U2FsdGVkX19tWrqQAHlg5vbg+s9hzijPzNmHjOr6iNevMtP2+mr75Zx51S1oCTnhSBpFC2VStwczIhfJa1H6uURgf2V5E08tQ7WGNe7nx80N/YPV99g51fIQxjvYjoqIPfGMZVGidch4W1VgVw0sCF5cmkNVWBS3jcf6PSEj8CXRigvdpU735uIGNw4sUqpgouhlrAkV41k0ZiVVSbLaOKw/7gg55K7wmYaXU3MoKnETq92RAY6wYUbDO49wvOJTPMnNsGBLbm8OYSWoLpTMzhhDs2xKhdJBONW6Lisdjb2sMySdfrXZMbVoF3mhKAjnvs44CPQon6viQOshoeqcGWGWdv3zeTN3Nyaj5ZPbsCUaEtm+BDT1zUTp3AyG4mecZ+CIIQr+PrED+udyXZao9eBT+ZRTcjB+2+aTNJimgAuxr28bf/A62I/YwooTs3Dn+j3El3a5eR1RKvU5JSDJRA5NPOA6AcW4OkX8VIeUZwipnLAQWMuCuK5Fnc4/zqO/9W7xnqel/ZdNq/vXjTmaaDsrAVIG20Gh9yboR0oPwnBHVmLKpkSAYUC1aSlFwyi9HKbfLJZ+HNdD/QsRyg0khnXnSWy+F4uefCnFE0L0bSi9zUPflFjQTn44Dc9dFG1JyB6nw16gLl3MH7x3pOKSnrbI7wze84kSnQZCdAbls9fQtKL73y40RQ/v87z5n4pBbIZN/ZNDEqr50I5SN+Q8LHwrY1FnbDVyhrJ1UIxUumcws/U8lsLt4ZUNWaggl8YApTuLVGAP3LyIbV8gP1H4IrytZKX24+GkDgzff04rP1orsnhgdbvEQZ+JA6M1CIjlv8aKGpYuq1CjqDVNjjkpAIHhMfdHIRO1F55fF4Z3cLlW0+ol1ifaUDPi3zrVxt/wLra43IdOKFO9n1hlUtuM4GchFRQRR+bQZdRBV04KIkEFVSg1Tlf0MLQR88cQpBBy8tEq4tOdru3eExPSwsU UtyguufU iJrPazFkD+8/QXm6nN6wSo9xB0A2ifnJfouvrAh+zAOhZAv9JTxo55F7qRd84KbHmjmBt4sUOAZKVLWE7EQTKJzUO7Oh9Mn8pXVadHTflgVvx63gyQ4cCDFDadE2tGxpeKcivXFmNMzBwc23NT6uUEtSjw1BhDqE82v6x+v6I7xpZF9hqlNmaK3qZyr9PSDcPB/qbwJmtAeCSoheQlFcqoInXj9yhDui+Y2Ix2aZGAe9Wur1C2PnGNrG3jt2MYcc97rNBjiLqecxiAxL4Tr1C20oMmuG3/4PoFcY+eYDGMJQ6ZKLAxfvN30HRQcE/jKo4bEdqJlomCkYFMhpM20LTOuY7CLbiCLR2FoJJZJvyhSMl5WbkPmrTc65jUg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Laurent Dufour Attempt VMA lock-based page fault handling first, and fall back to the existing mmap_lock-based handling if that fails. Copied from "x86/mm: try VMA lock-based page fault handling first" Signed-off-by: Laurent Dufour Signed-off-by: Suren Baghdasaryan --- arch/powerpc/mm/fault.c | 41 ++++++++++++++++++++++++++ arch/powerpc/platforms/powernv/Kconfig | 1 + arch/powerpc/platforms/pseries/Kconfig | 1 + 3 files changed, 43 insertions(+) diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 2bef19cc1b98..f92f8956d5f2 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -469,6 +469,44 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, if (is_exec) flags |= FAULT_FLAG_INSTRUCTION; +#ifdef CONFIG_PER_VMA_LOCK + if (!(flags & FAULT_FLAG_USER) || atomic_read(&mm->mm_users) == 1) + goto lock_mmap; + + vma = lock_vma_under_rcu(mm, address); + if (!vma) + goto lock_mmap; + + if (unlikely(access_pkey_error(is_write, is_exec, + (error_code & DSISR_KEYFAULT), vma))) { + int rc = bad_access_pkey(regs, address, vma); + + vma_read_unlock(vma); + return rc; + } + + if (unlikely(access_error(is_write, is_exec, vma))) { + int rc = bad_access(regs, address); + + vma_read_unlock(vma); + return rc; + } + + fault = handle_mm_fault(vma, address, flags | FAULT_FLAG_VMA_LOCK, regs); + vma_read_unlock(vma); + + if (!(fault & VM_FAULT_RETRY)) { + count_vm_vma_lock_event(VMA_LOCK_SUCCESS); + goto done; + } + count_vm_vma_lock_event(VMA_LOCK_RETRY); + + if (fault_signal_pending(fault, regs)) + return user_mode(regs) ? 0 : SIGBUS; + +lock_mmap: +#endif /* CONFIG_PER_VMA_LOCK */ + /* When running in the kernel we expect faults to occur only to * addresses in user space. All other faults represent errors in the * kernel and should generate an OOPS. Unfortunately, in the case of an @@ -545,6 +583,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsigned long address, mmap_read_unlock(current->mm); +#ifdef CONFIG_PER_VMA_LOCK +done: +#endif if (unlikely(fault & VM_FAULT_ERROR)) return mm_fault_error(regs, address, fault); diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig index ae248a161b43..70a46acc70d6 100644 --- a/arch/powerpc/platforms/powernv/Kconfig +++ b/arch/powerpc/platforms/powernv/Kconfig @@ -16,6 +16,7 @@ config PPC_POWERNV select PPC_DOORBELL select MMU_NOTIFIER select FORCE_SMP + select ARCH_SUPPORTS_PER_VMA_LOCK default y config OPAL_PRD diff --git a/arch/powerpc/platforms/pseries/Kconfig b/arch/powerpc/platforms/pseries/Kconfig index a3b4d99567cb..e036a04ff1ca 100644 --- a/arch/powerpc/platforms/pseries/Kconfig +++ b/arch/powerpc/platforms/pseries/Kconfig @@ -21,6 +21,7 @@ config PPC_PSERIES select HOTPLUG_CPU select FORCE_SMP select SWIOTLB + select ARCH_SUPPORTS_PER_VMA_LOCK default y config PARAVIRT From patchwork Mon Jan 9 20:53:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAB8CC54EBD for ; Mon, 9 Jan 2023 20:55:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 48EAE900004; Mon, 9 Jan 2023 15:55:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 43E7D900002; Mon, 9 Jan 2023 15:55:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3066E900004; Mon, 9 Jan 2023 15:55:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1DA83900002 for ; Mon, 9 Jan 2023 15:55:16 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F0AEEAB133 for ; Mon, 9 Jan 2023 20:55:15 +0000 (UTC) X-FDA: 80336465790.28.92C6F0E Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf06.hostedemail.com (Postfix) with ESMTP id 50A16180007 for ; Mon, 9 Jan 2023 20:55:14 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=C0RC5qgB; spf=pass (imf06.hostedemail.com: domain of 3MX-8YwYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3MX-8YwYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297714; a=rsa-sha256; cv=none; b=IYcLkIT+BT78T0SZABdHVN8EbMLksNXsxiX4GASqyVN2gx/JB+0N/lcjJBsdZQVaJeAe4n G3FGs9FTn9W7ZNx0uesYvz7TN5F5Rg7u2gsZORxEKX6uqdKw0TDzPN7ho24x6m7K7S806G J2GyTNyGd50kgkD+ngJOMfFX3D7nLQw= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=C0RC5qgB; spf=pass (imf06.hostedemail.com: domain of 3MX-8YwYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3MX-8YwYKCEY02zmvjowwotm.kwutqv25-uus3iks.wzo@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297714; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JqE4Jv/Ie7jJyt+ylXoJ6xWWNYJ7mAmWvaygaHhGgmA=; b=4sNE5bxT6xmZBUK9aNMQlc72TJTBSG9uLtZJhiidFiKh1xZRpl24MK6UtJREHil5kLW2bj J+e7mSz595r98+09MCTH2q18Luh6UTn+FJ6aemyD2Q7jiAl61DUOzmElWzd8WVAmX/P8Sx 7LAnuBEl9jxLU+i/CRmZRHwPzrPPV6E= Received: by mail-yb1-f201.google.com with SMTP id z17-20020a25e311000000b00719e04e59e1so10334040ybd.10 for ; Mon, 09 Jan 2023 12:55:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JqE4Jv/Ie7jJyt+ylXoJ6xWWNYJ7mAmWvaygaHhGgmA=; b=C0RC5qgBD/V26736LQT1/agfFQE3oawoBsVodaZr+HfHKR+SKXSkxzHnqRRZUQyjCF /VKZZZ+vJ6kaEdIfo26R2t3ZnpRNnactrnv5DhopT9zYmpMMrFjwmeMN0NuvQGOG9Ma5 xgohCqYaIMlqK/c6f2BeWoXmU7iZ4/9gccjpi+n1Vpya+2P9xxgUSOybhe5uijKT6IMh I/M50QEDZcxaCjlrD2td9ExwPf2mPPj9P3UPSnAgQR/cCy9DPbczOjnWTPsq0XpCcdbi g/lQrqmdlC6kTELYjuH3lVXIdNQlERXCKK1DKiV3+4JsEhkwrKU78EAKu+F90rDI8Z8t c6mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JqE4Jv/Ie7jJyt+ylXoJ6xWWNYJ7mAmWvaygaHhGgmA=; b=hrR21YQk+WRJ0X9B3kQXrySMPO8UGPZJgz0bHNXg3hnCBsQzZIGsKvD4ZAQTTBmM6y Il+H2wdFJpANTbJtBaOL2tzbSqjeTLxWG7PMa0MWUhc2DATeLA4g0KIJir07REwfvA41 nYZ8DglFjN8sNSCcpqTSsuopYKUqvvTeu+q821lpo4Rusorw+QzKsIDZhNAHNWm5MVNn MCtNn3CmxlRu3iBqkZqdrjJ5e+ZHzcrNbSYWwFia3GL6HHcx+nXRh/KtDmlMyzxNc0fU Gucd8mX5lLIIhv9ZJbr0xQ2g7B9m9XSeSRF+A74+sOvfQlV74xWNvY9djV+ly4GJ8q8i EauQ== X-Gm-Message-State: AFqh2kqZ5OuXwPaB5icBCn2g6RNLIlLfLhZTBD0+vn41Dv0bTMaAc15/ N9BvLAxryoApAOZrno2t441lFy1ZtBs= X-Google-Smtp-Source: AMrXdXsVSd5Wz07hI5hxUAzF2wzs/BFzwaLObvOXI24uMhgcUHGkQCCTKMo/jOVrGruw6pJEAc/B3Pke/WM= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a0d:dbce:0:b0:4ad:7104:1f63 with SMTP id d197-20020a0ddbce000000b004ad71041f63mr3427748ywe.49.1673297713421; Mon, 09 Jan 2023 12:55:13 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:32 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-38-surenb@google.com> Subject: [PATCH 37/41] mm: introduce mod_vm_flags_nolock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 50A16180007 X-Stat-Signature: 6xhgh6ou3ramcwyfaw3obis4um1p8q74 X-HE-Tag: 1673297714-479492 X-HE-Meta: U2FsdGVkX1+2bWq6dmtUkd9CR8tRNgziXymSJXlUBR9QsUuXedQkAdGvlyc5b2xFWGSfEX5gHuzSk9IlgvRWmsqCvew/uHAQen3K+CDoRH5LQYSmBQK0BSVORiox3x692ArKYZ8EB3aUZZSCnbCT3Xy0rDW7Jct+BSI6/vK5cExHQHBKlcXcQJ83yH+qVJf9LnAtSlCxnTvL2HPEQ898ipbuqpZuibm3V0gHEaxADsaF5d3T4UxWYcwflxUlHw0mB6NH/IqITlxrhRPplh0rOzFHYtOSDzbxacj6Ljz0NaqRDe1vhJ1J/haWA2V3rSf+n7wHP2qkWTUVQd2TFEqKFrAtK3Lf20h9UqrxZ4p+IcIizr8PCLwYM6FR09r2gVHGmYUtC/kyaUW39a51BcWjSJuH2zs29jIarM38SFz/6hPAiLvqmjuhEC1wEjxRdpLQOocbCU6UF6stwrZ+/NYLdsP82Nos7idStgUQUtLGvXEhtLmmU8k7dvim1N2+9mksdp2r1cw8T9kJFbLuSSTU59MKoVsL1rNmi/VPv5pmKOs2f4KIfwPNwF2aMnMtoeEoSICLDVDU36NuVeTgb/i9F9kZwIBPmp7G9CvapCgQQyobXUPzRSNOXfvqBUh7KY22eTYl/syTM4JHqZ932c3CIoa6BEDGPKhxtWAKszc6vJoXTLJVIKaQqhjK9uuCTswMOM9H9FRcLKwNAX8bqZ0rGwHzgFJgRj0TSaF53tuoarsta2gJFfogPCVquJGLX+r2kDvDVkinu9uYVz41dcWsHQRQEuYmKqpUNqyurHKJdOf4CWzQtDburKpjRzn25hnc5Fe5E2T1MXhXBe81zanq/ouTYHL0hIgNvAU6RtBpXLojCTM4TUT43yVvnkp4DRKraLL9SAj2tfAiMCSDWuRsKgrFWDU2ncJiPJz+HBUyz/a+YScZ0wELW+DvU2NiuX5xl+hmQGpOqcEUvXKlfw9 o7ejGGiJ cVYFqbDKCi+Xs+kh0tr5GBmhVK6MC2i5vkY92LjR8Lcnd7+x17eAKruvxW6JOh+/El8YgLXryRz+YljEf+7SaRLOzduX+Zi19Sy1Lb9lq6ZI1oIKHR48SukKHC39Pe50GnJa2TMUVk+hjstJrDibz4BXVRLxsTUD+Vnf80z0fKQ3vHtASQ8kVb0ef6A0CgSlnB00RKItZXuAKbHJVodAPegKUd1Vd0Kp1WAbjZTslVU2ZokgSz21KENvYA6/3h5V4GSE1dTkmpnugvisTOlCscfqQLajgda6++Iz1uuHHYc9Z8bxOX9U2hVRRI0CUiip55W8aMR7poqRgtncCwmRFdNn3gt9/cEqJ99kNyNjSlOsh9eF6wd9KY38iQlds4+k7Wry2SbGTsSPCjjVNpRU1o5R8+dE9mBI0dTGn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000023, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In cases when VMA flags are modified after VMA was isolated and mmap_lock was downgraded, flags modifications do not require per-VMA locking and an attempt to lock the VMA would result in an assertion because mmap write lock is not held. Introduce mod_vm_flags_nolock to be used in such situation. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 2e3be1d45371..7d436a5027cc 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -743,6 +743,14 @@ void clear_vm_flags(struct vm_area_struct *vma, unsigned long flags) vma->vm_flags &= ~flags; } +static inline +void mod_vm_flags_nolock(struct vm_area_struct *vma, + unsigned long set, unsigned long clear) +{ + vma->vm_flags |= set; + vma->vm_flags &= ~clear; +} + static inline void mod_vm_flags(struct vm_area_struct *vma, unsigned long set, unsigned long clear) From patchwork Mon Jan 9 20:53:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F6DAC5479D for ; Mon, 9 Jan 2023 20:55:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD094900002; Mon, 9 Jan 2023 15:55:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A8038940007; Mon, 9 Jan 2023 15:55:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D278900005; Mon, 9 Jan 2023 15:55:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 7BF83900002 for ; Mon, 9 Jan 2023 15:55:19 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4E8B6AB14C for ; Mon, 9 Jan 2023 20:55:19 +0000 (UTC) X-FDA: 80336465958.06.93DB5E7 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf30.hostedemail.com (Postfix) with ESMTP id AA87F8000A for ; Mon, 9 Jan 2023 20:55:17 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=dIxUWUkr; spf=pass (imf30.hostedemail.com: domain of 3NH-8YwYKCEk352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3NH-8YwYKCEk352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297717; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iUx/OB3iQQFsS6Pcn0sWsw86mSb+m9wyFp0pF/RbXSk=; b=1BdUxbLLBpgUGJZUl7TFRwTkCvfkyXCHpEBeFzqS7r9wfUyqYhVPXtM59q8luFOzU55R9Q oMaRbziuSJllccmgnZ+/nFAO3GiQ05JAdMYyTgx76J0C9Yzugjm96V6bvab7dVWjGSYDtt Vo1ODNEl4SrhvrLZUnxfxsws5a5Z3MQ= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=dIxUWUkr; spf=pass (imf30.hostedemail.com: domain of 3NH-8YwYKCEk352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3NH-8YwYKCEk352pymrzzrwp.nzxwty58-xxv6lnv.z2r@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297717; a=rsa-sha256; cv=none; b=Jy1al1TxdJ80RQW55pEIP9phvXQ8+EjTX+P7hFCyBRyIFCAN768qZPCmUi32iVvgK9WUdq iUar479Slkqy1kd1d7uWZfCzvgzbunVCTUMZI+6bWhxB+auYC/PLEwJ73m2EUgL6whYKWc G675dV095i1pAIP5KNuiQjQx/KlV/FE= Received: by mail-pl1-f202.google.com with SMTP id u2-20020a17090341c200b00192bc565119so7008259ple.16 for ; Mon, 09 Jan 2023 12:55:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iUx/OB3iQQFsS6Pcn0sWsw86mSb+m9wyFp0pF/RbXSk=; b=dIxUWUkrPawF5nCoP+HUjvA+1vmI2x8utDxJKsWUDKbZyEnrEFBNIhzaM8Fi8ju0R6 eFijpcCpNnzwFYn4qQcfR/+HsKpxnsThGu7XxjIktqkOlh8pbTgjPjgxlF3er9jtpiFK 3GSfGJF3spx9qr3lx0Xbz1tHEn1iO2FRbitd8mVM168Ji21A6WJbwvjHXFAHlbmszk/z PHIXCswYcUFMYiFzIipRXQxj+Lq9k2DTCyxkIwApZosMOj5/aAMGnoES19O/qzXO1kO2 XId6gaep+ySgAUv8E1eSLybQz1XP2tMRZPlcvFIYudobcMhpLFz5bkGWnIsnoWm/3FYR M9Mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iUx/OB3iQQFsS6Pcn0sWsw86mSb+m9wyFp0pF/RbXSk=; b=LdUtUDnV/bUGLPLaI7x8ALjSHJulVkBYxx+Q6QIuW2CBX3yyqQGDa+EnK5k9U6UIkT zTvEZVNhkut0Q6Cm+VEv+au1hCip1L1WumZoXW2P2wdchM8hZxI6sI5EzF6R2JYDoDBv uQauHHycVAA8/mroWnOaTvb9+3/3W4WqXb9DAv9clp+phP6kENBfHtcr9X89OsVRAE/Z 8LaXwfmofn8IhKwLoJls15MWjUd6h/5TVvbHj5Pj3wx7V4tnT+IzbmN2bBpKfGaPwYwF ex8WReydwXtK5T/CNDk80sOdtkTPBSZW8WkY/lREjUc1gyomE/9dElXLImtJJYQm2O2d f88A== X-Gm-Message-State: AFqh2krGRE4/wYDTyghBabY/W6mQaMszguNkobHEgc2fEKPqDSRCyF3o UsL4l3jm3RX3NnSUyIcAdO11Ydr0qOE= X-Google-Smtp-Source: AMrXdXtKEO5NZ0LV26PgvCCbCCrX8iN+m+pnC68agl9KzwVT0TYKaTzsINmDv+j62IKmNfwju5kMUEnv/YA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a17:90b:370a:b0:226:f53b:d370 with SMTP id mg10-20020a17090b370a00b00226f53bd370mr875190pjb.75.1673297716576; Mon, 09 Jan 2023 12:55:16 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:33 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-39-surenb@google.com> Subject: [PATCH 38/41] mm: avoid assertion in untrack_pfn From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspamd-Queue-Id: AA87F8000A X-Stat-Signature: 34qxcrf1f8bazfct981qk46536o1fr1w X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1673297717-178467 X-HE-Meta: U2FsdGVkX1/QbNFADeujAeArGo3GQsmWk1MQUCwLnM4gFB658xTrGRS+5xzsNQG+iJiGi0qVcmfw5+xPCi5a6yiyx8EgZDGOywxfKG9oZah77qdywiHby+glsGc6qk1+MkOUO8fTEKxzLbP4Tck3KNRZWQUaCxnvuCBQUcl/ATpowOoRiPk2qOQe9vfA8pIjMgDWX1nebL8lb1FIGTUG81rl0L5V9LeoNm9uKZQ2LJlD5bSNuWdXBX0Wfey0F02EGL2whPeaMPjOctJv+UAGcz9cDK9PMT07cxqm2r8v6eBDUbGY9bkfaMhgAiaA1Fu7ZzuzIw6hZgY2UKL2y+oHToAbQHwmIxb1NMmk+5DIOfpecsJkICytdRP3shvIAkem21Wlr0H2e3EGP6d5DAM0YVIFp+W7j33VIIguhacHLKefBFsZIKLCWtH1xUCPkSVISGrgP+TP6xqN65p2EdNTzzs/CdBg8+LJEopxjEZgils+sUk4kllLybTvhm3u6KbqfvKvS25uTOPJgYL+UdxhgrhwkafW5rzToxvW8tEAao88zn3Lyg/EPohrjb5pMHuad8fQ0UXTiP6bT0NqoM9XVpvw1Il//T/yClNCQrHaE5uCEBrK9fGH6dXriL5qNXXZK8dsHPfWqZWcRYB/3AVWLAiIMqzn5vnv8tXxia51Zhn8Io1w1BOljpXeyonVSow88fHq9oZoBqQL5slvgM20bAkW+abkcjX5enisDOUVPn7UGaheih8WqpdcQalt2u3VjdYfh0WleaFMRhGecwbhtZdrfzvc2gF4LKb5NgBEsOEtSZGVG6cPqJqaFOcoOCzR1yQiwUZ+ZcIgIUVixhpIlre08QS9w/eEXm57TiqFNJx7AvptomQdh0cT1xcdzkFP+IR8vw71NUjLlgeQAGWno0//GLTxVim7REK+FDV2Cxz7IBxhhtFeeXMOVLjfM1em2xA4CgwguaKPnABeGlz MhgjGMVT Wwu9nrJlWMHcLMjiiSAFL7VnH/+QHt+mEk0SZhjwu3I29mu548Q1BDjntfaSbyBWpwefvJ7b24sYmPxI+dmo2MqFI6WXw7+a5wdwbb+rqno0ZRV3iOAdP6n0WjyfVqayLX8kL0qJocbOCh5M5OQh8iXI0NHOn+xPWy5tRuumwb1pKdHgGoISPdqWnIiSUJ23kLV67W0TqRbjwZb5vOeMVfJpynD7lGandH31FISn+E+1dBLhrm6dCgVrupE4UMbpxDczoejzgzmtUOnjL7Gh+W/EdwvdEX4KB0bz8ZqBB5cHh1czdheS9jaRO3cxlOwGlmK0PZsUgpEsyRgrbceSsf2j2PZU/3/cYO+6YALoJsi8rJ2xK6QwkzZB0LrF/xPin/vDRv7VUB2axYyvEM9Yftr1MZfqhQDwUP5Mq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: untrack_pfn can be called after VMA was isolated and mmap_lock downgraded. An attempt to lock affected VMA would cause an assertion, therefore use mod_vm_flags_nolock in such situations. Signed-off-by: Suren Baghdasaryan --- arch/x86/mm/pat/memtype.c | 10 +++++++--- include/linux/mm.h | 2 +- include/linux/pgtable.h | 5 +++-- mm/memory.c | 15 ++++++++------- mm/memremap.c | 4 ++-- mm/mmap.c | 4 ++-- 6 files changed, 23 insertions(+), 17 deletions(-) diff --git a/arch/x86/mm/pat/memtype.c b/arch/x86/mm/pat/memtype.c index 9e490a372896..f71c8381430b 100644 --- a/arch/x86/mm/pat/memtype.c +++ b/arch/x86/mm/pat/memtype.c @@ -1045,7 +1045,7 @@ void track_pfn_insert(struct vm_area_struct *vma, pgprot_t *prot, pfn_t pfn) * can be for the entire vma (in which case pfn, size are zero). */ void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, - unsigned long size) + unsigned long size, bool lock_vma) { resource_size_t paddr; unsigned long prot; @@ -1064,8 +1064,12 @@ void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, size = vma->vm_end - vma->vm_start; } free_pfn_range(paddr, size); - if (vma) - clear_vm_flags(vma, VM_PAT); + if (vma) { + if (lock_vma) + clear_vm_flags(vma, VM_PAT); + else + mod_vm_flags_nolock(vma, 0, VM_PAT); + } } /* diff --git a/include/linux/mm.h b/include/linux/mm.h index 7d436a5027cc..3158f33e268c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2135,7 +2135,7 @@ void zap_page_range_single(struct vm_area_struct *vma, unsigned long address, unsigned long size, struct zap_details *details); void unmap_vmas(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *start_vma, unsigned long start, - unsigned long end); + unsigned long end, bool lock_vma); struct mmu_notifier_range; diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 1159b25b0542..eaa831bd675d 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1214,7 +1214,8 @@ static inline int track_pfn_copy(struct vm_area_struct *vma) * can be for the entire vma (in which case pfn, size are zero). */ static inline void untrack_pfn(struct vm_area_struct *vma, - unsigned long pfn, unsigned long size) + unsigned long pfn, unsigned long size, + bool lock_vma) { } @@ -1232,7 +1233,7 @@ extern void track_pfn_insert(struct vm_area_struct *vma, pgprot_t *prot, pfn_t pfn); extern int track_pfn_copy(struct vm_area_struct *vma); extern void untrack_pfn(struct vm_area_struct *vma, unsigned long pfn, - unsigned long size); + unsigned long size, bool lock_vma); extern void untrack_pfn_moved(struct vm_area_struct *vma); #endif diff --git a/mm/memory.c b/mm/memory.c index 12508f4d845a..5c7d5eaa60d8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1610,7 +1610,7 @@ void unmap_page_range(struct mmu_gather *tlb, static void unmap_single_vma(struct mmu_gather *tlb, struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, - struct zap_details *details) + struct zap_details *details, bool lock_vma) { unsigned long start = max(vma->vm_start, start_addr); unsigned long end; @@ -1625,7 +1625,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, uprobe_munmap(vma, start, end); if (unlikely(vma->vm_flags & VM_PFNMAP)) - untrack_pfn(vma, 0, 0); + untrack_pfn(vma, 0, 0, lock_vma); if (start != end) { if (unlikely(is_vm_hugetlb_page(vma))) { @@ -1672,7 +1672,7 @@ static void unmap_single_vma(struct mmu_gather *tlb, */ void unmap_vmas(struct mmu_gather *tlb, struct maple_tree *mt, struct vm_area_struct *vma, unsigned long start_addr, - unsigned long end_addr) + unsigned long end_addr, bool lock_vma) { struct mmu_notifier_range range; struct zap_details details = { @@ -1686,7 +1686,8 @@ void unmap_vmas(struct mmu_gather *tlb, struct maple_tree *mt, start_addr, end_addr); mmu_notifier_invalidate_range_start(&range); do { - unmap_single_vma(tlb, vma, start_addr, end_addr, &details); + unmap_single_vma(tlb, vma, start_addr, end_addr, &details, + lock_vma); } while ((vma = mas_find(&mas, end_addr - 1)) != NULL); mmu_notifier_invalidate_range_end(&range); } @@ -1715,7 +1716,7 @@ void zap_page_range(struct vm_area_struct *vma, unsigned long start, update_hiwater_rss(vma->vm_mm); mmu_notifier_invalidate_range_start(&range); do { - unmap_single_vma(&tlb, vma, start, range.end, NULL); + unmap_single_vma(&tlb, vma, start, range.end, NULL, false); } while ((vma = mas_find(&mas, end - 1)) != NULL); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); @@ -1750,7 +1751,7 @@ void zap_page_range_single(struct vm_area_struct *vma, unsigned long address, * unmap 'address-end' not 'range.start-range.end' as range * could have been expanded for hugetlb pmd sharing. */ - unmap_single_vma(&tlb, vma, address, end, details); + unmap_single_vma(&tlb, vma, address, end, details, false); mmu_notifier_invalidate_range_end(&range); tlb_finish_mmu(&tlb); } @@ -2519,7 +2520,7 @@ int remap_pfn_range(struct vm_area_struct *vma, unsigned long addr, err = remap_pfn_range_notrack(vma, addr, pfn, size, prot); if (err) - untrack_pfn(vma, pfn, PAGE_ALIGN(size)); + untrack_pfn(vma, pfn, PAGE_ALIGN(size), true); return err; } EXPORT_SYMBOL(remap_pfn_range); diff --git a/mm/memremap.c b/mm/memremap.c index 08cbf54fe037..2f88f43d4a01 100644 --- a/mm/memremap.c +++ b/mm/memremap.c @@ -129,7 +129,7 @@ static void pageunmap_range(struct dev_pagemap *pgmap, int range_id) } mem_hotplug_done(); - untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range)); + untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range), true); pgmap_array_delete(range); } @@ -276,7 +276,7 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, if (!is_private) kasan_remove_zero_shadow(__va(range->start), range_len(range)); err_kasan: - untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range)); + untrack_pfn(NULL, PHYS_PFN(range->start), range_len(range), true); err_pfn_remap: pgmap_array_delete(range); return error; diff --git a/mm/mmap.c b/mm/mmap.c index a256deca0bc0..332af383f7cd 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -2209,7 +2209,7 @@ static void unmap_region(struct mm_struct *mm, struct maple_tree *mt, lru_add_drain(); tlb_gather_mmu(&tlb, mm); update_hiwater_rss(mm); - unmap_vmas(&tlb, mt, vma, start, end); + unmap_vmas(&tlb, mt, vma, start, end, lock_vma); free_pgtables(&tlb, mt, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS, next ? next->vm_start : USER_PGTABLES_CEILING, lock_vma); @@ -3127,7 +3127,7 @@ void exit_mmap(struct mm_struct *mm) tlb_gather_mmu_fullmm(&tlb, mm); /* update_hiwater_rss(mm) here? but nobody should be looking */ /* Use ULONG_MAX here to ensure all VMAs in the mm are unmapped */ - unmap_vmas(&tlb, &mm->mm_mt, vma, 0, ULONG_MAX); + unmap_vmas(&tlb, &mm->mm_mt, vma, 0, ULONG_MAX, false); mmap_read_unlock(mm); /* From patchwork Mon Jan 9 20:53:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 679C0C61DB3 for ; Mon, 9 Jan 2023 20:55:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 049518E0006; Mon, 9 Jan 2023 15:55:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F3BA3900005; Mon, 9 Jan 2023 15:55:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8E07900006; Mon, 9 Jan 2023 15:55:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C61A9900005 for ; Mon, 9 Jan 2023 15:55:21 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 681BCC0F43 for ; Mon, 9 Jan 2023 20:55:21 +0000 (UTC) X-FDA: 80336466042.22.D868506 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf11.hostedemail.com (Postfix) with ESMTP id CC06D4000A for ; Mon, 9 Jan 2023 20:55:19 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="R+wIE/Pt"; spf=pass (imf11.hostedemail.com: domain of 3Nn-8YwYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Nn-8YwYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297719; a=rsa-sha256; cv=none; b=y7AHMZ/BRTvJFFFurJAAc0vr7sKjyAS+gp2lLmKeqh0/Row/oFnDyiAGmltz7g+Z64BMQR ohqjElTEeq1/1q9P/dKIKkWCdK8KJ78rBuAaMM47JX+hmVU0Gk7zCxkbcFRMh/8rJ84sLm dNoUUecW1uBNUtjBVXTzY2OlodMx/lM= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b="R+wIE/Pt"; spf=pass (imf11.hostedemail.com: domain of 3Nn-8YwYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3Nn-8YwYKCEs574r0ot11tyr.p1zyv07A-zzx8npx.14t@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297719; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q4uicoifY7gC/4yqte16SYBfutCg5tFbw97tUXlS1Bs=; b=xo22qUWmeBakmVi/TM5stdxNPP6JX27yvwd32ajWgUEz9ldSacWUFTG9Pl3TilY4923nAv M41kEKOBmb2o0lbGN6x2T2cvoTVstQXJHkzy3ulJaBsF4BTj8zPhx70O/krEMJaHIh6nUL mw6Fi6YaRMRDCgueQZog1DJpXLHxpvI= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4c0fe6e3f13so104583667b3.0 for ; Mon, 09 Jan 2023 12:55:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=q4uicoifY7gC/4yqte16SYBfutCg5tFbw97tUXlS1Bs=; b=R+wIE/Pt4KhRz8VFgR0zMOS2JkvFmXy26q+OSQ9LjCyq0QISjHNPW3JBEjDsyfPMZn JHIhnIofZ7qGF96jAqweLgH0bhrxbWlUBYN16b0h3IUNriVtZxEflnZ/KJaCsGgJuROJ 7NtSec9kWHRW8dxlpNmOxFIq510sDiRPIwm32r6PlWpXwb1OQmeULeUN7CB+D4fk8aTq 1FAeEWt3FnKxXn+FEK6NZ2a1Nn+94uGmt6MDM7k61uLqgDX1MFlIF8o4Zu3p/TF9jRMz yr+WOD/8oTTja8WHghFFUJvljuYoTFD7DGiXYFX6wrr/dkh/UC1lDtuKJ2cCE65save6 YCEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=q4uicoifY7gC/4yqte16SYBfutCg5tFbw97tUXlS1Bs=; b=SFjYZvc+0MCcClOKC5Oz3+c+x0SI3yu7/JGSk12YheG9knLXnR7j8W8ifZXSxaA3OO aX8xsfXQfWcH/uweeQy4x03zQiDM4WEGtybENw6fdRgY8Yb6NaSkviNQxot+CGIdKcv2 CwToUELDbVb/Y14Rb+CgDZ7sxTzQWFstgah28bwwJGFrtUAPupG9Yi+qCJZAFHdDHxSI cd9whI1X1perSdgQuyTv5k0UdaRmCNieKqhZh0OCE8UXR/t0lqFKFTmPWBCMXL3fbrUh 4Zv9/fjww2NK0vpHv8fIKTTpNpxdqePj34moIuaHuCp1RGPlrCJjSXc5ArKeTNm1tveI StFQ== X-Gm-Message-State: AFqh2kqtdMEGTZuIoU6zGhMV1FJkeWxXBLqodhI/a0veosPYxfa4K18J nSstWHqkkk5Vi0gIyvOXZVlwEi2zg3M= X-Google-Smtp-Source: AMrXdXufHU+YEXa/HPbI4vLLyfNw163JJwFMEsJciRnPr1X88dc4ccEYYlMB1fPlEQ5O4fkaQPa8neGtLug= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:9246:0:b0:6f2:2d54:dcdf with SMTP id e6-20020a259246000000b006f22d54dcdfmr6080399ybo.144.1673297718997; Mon, 09 Jan 2023 12:55:18 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:34 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-40-surenb@google.com> Subject: [PATCH 39/41] kernel/fork: throttle call_rcu() calls in vm_area_free From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: CC06D4000A X-Stat-Signature: 6hya973xb37yach99j6ace4tk4jg8r9m X-HE-Tag: 1673297719-329574 X-HE-Meta: U2FsdGVkX19Dw7DKMRSs/sLasA3xLCeHY03y0upEs6zC9aE724O0+RrvvkrR0vdpuOIClgNflZDUtqoz6CsDp/XUBrWn/a1jOpnEyJXs1F0nNPPxZOkHeu2w4KIA+O2W51cnrUDe0G8kIX8se61zMsfrNBdNXo2LJLcLyRtasA5A31DFlgs/qJfgkb1ra93rXpppVUz5JZVps0S1zzHG90BqkC40oNtQEliUk4/yGIlZhJxdStUq4h+6EwZcm8PSYKmPm2C7LKoicAg00PGHtPEm568mn1ToEH5HG7ppQoyEEGs6mrXa8V/g1xiNZJwh+8O3lvrLriV4d83JlpVuh3DycZ5iKPjLSSQpR/ZOfUzRrBb/J+HOqB0nlZCULUiccNFN3XjI8z79/TsBSa2FtQlQKb73slF3Lowr96InjdmwcBOGH3yP2AwpVVe2aJR2rd3NOv5iTMCSKd89u4fDxP9LAUXz1RmmQMfUWqogWB0ueZ48BowdMs8Mzh6aIBmdzactrZL3F13pgCbs45t0f/hcP8KEsYIOt48jiOJqDqkruMDP6OrXvP4mY7hhajmkPVM24Ssy1E3ev14ElQ4w0nHbi3/vaJXFkmDSDd/rMbKRntEKt8Jt3Giv0YGtpKLOjPlAYpGoKV8mA0gXarKq+YgcsrxF5n1Ja97msnWWzzwG8YPBgWTn/vH9Ap7022t7/OH3DmHYIRseB0ioF98v+qpdZ3en0sEsFS7itxWCGBJ0XdaFJtq9uq9pKSmtfNft6xi4VXCHAzt5j9ITrRjHEWhBgilo+rgMxPZpvNldmzbc1Qq+nV/vokf9wCW/nrJEm7cXOQEx3QyTk/QdX6WDbru2lh7AosT5zrilO39f8WnyHJO9VKTUFp0C/aLm6zSKVVfUoVDCKK1mnv0MPFx54VrsHPbcAfE1qLDl1KOJwToaUkjpdKunoQpT1T3QDM2vsTpGvpLEpNOPgHa3GSm 6YGaYzFC rWt0PMfJphCnsEh2feQWpoB24ydjvOey72wZnGTF5eIn0uBvNMqmnleE2Q8z1NKfo3BS6Kc10bLINBBD3yOcKo1bN4gicDLJBf4i+PG+6Qy7qLUgVv7ical2/SBQ/pgU4txCKboUmlTuSbXvhbot9INOEB0L1ghqysz+GJS923INoWL+tEYlx0/gALQ+XML53BLxHA2sSkYzdEyulO3rQE1s+g0e1twzF8augKE9amd8LeGeZTsbSFVuCK7ZFyQo9D2TLgAWF1OjOdC46S2Abvj4QGlQrf4tXAKS0hrjtPnVnPl+oa6bhV8Lya3qiuI81n6gsVFp/C7mvQciZCHoS53IGzdlLiwJr89JcmG+G8Ltu13lNX94VIC5JgnrLYNmb2VWJFx8zktlm+wlQ1zzFCmSMiRf8l/29TOOI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: call_rcu() can take a long time when callback offloading is enabled. Its use in the vm_area_free can cause regressions in the exit path when multiple VMAs are being freed. To minimize that impact, place VMAs into a list and free them in groups using one call_rcu() call per group. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 1 + include/linux/mm_types.h | 19 +++++++++-- kernel/fork.c | 68 +++++++++++++++++++++++++++++++++++----- mm/init-mm.c | 3 ++ mm/mmap.c | 1 + 5 files changed, 82 insertions(+), 10 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 3158f33e268c..50c7a6dd9c7a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -250,6 +250,7 @@ void setup_initial_init_mm(void *start_code, void *end_code, struct vm_area_struct *vm_area_alloc(struct mm_struct *); struct vm_area_struct *vm_area_dup(struct vm_area_struct *); void vm_area_free(struct vm_area_struct *); +void drain_free_vmas(struct mm_struct *mm); #ifndef CONFIG_MMU extern struct rb_root nommu_region_tree; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index fce9113d979c..c0e6c8e4700b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -592,8 +592,18 @@ struct vm_area_struct { /* Information about our backing store: */ unsigned long vm_pgoff; /* Offset (within vm_file) in PAGE_SIZE units */ - struct file * vm_file; /* File we map to (can be NULL). */ - void * vm_private_data; /* was vm_pte (shared mem) */ + union { + struct { + /* File we map to (can be NULL). */ + struct file *vm_file; + + /* was vm_pte (shared mem) */ + void *vm_private_data; + }; +#ifdef CONFIG_PER_VMA_LOCK + struct list_head vm_free_list; +#endif + }; #ifdef CONFIG_ANON_VMA_NAME /* @@ -693,6 +703,11 @@ struct mm_struct { */ #ifdef CONFIG_PER_VMA_LOCK int mm_lock_seq; + struct { + struct list_head head; + spinlock_t lock; + int size; + } vma_free_list; #endif diff --git a/kernel/fork.c b/kernel/fork.c index 6d9f14e55ecf..97f2b751f88d 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -481,26 +481,75 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) } #ifdef CONFIG_PER_VMA_LOCK -static void __vm_area_free(struct rcu_head *head) +static inline void __vm_area_free(struct vm_area_struct *vma) { - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); /* The vma should either have no lock holders or be write-locked. */ vma_assert_no_reader(vma); kmem_cache_free(vm_area_cachep, vma); } -#endif + +static void vma_free_rcu_callback(struct rcu_head *head) +{ + struct vm_area_struct *first_vma; + struct vm_area_struct *vma, *vma2; + + first_vma = container_of(head, struct vm_area_struct, vm_rcu); + list_for_each_entry_safe(vma, vma2, &first_vma->vm_free_list, vm_free_list) + __vm_area_free(vma); + __vm_area_free(first_vma); +} + +void drain_free_vmas(struct mm_struct *mm) +{ + struct vm_area_struct *first_vma; + LIST_HEAD(to_destroy); + + spin_lock(&mm->vma_free_list.lock); + list_splice_init(&mm->vma_free_list.head, &to_destroy); + mm->vma_free_list.size = 0; + spin_unlock(&mm->vma_free_list.lock); + + if (list_empty(&to_destroy)) + return; + + first_vma = list_first_entry(&to_destroy, struct vm_area_struct, vm_free_list); + /* Remove the head which is allocated on the stack */ + list_del(&to_destroy); + + call_rcu(&first_vma->vm_rcu, vma_free_rcu_callback); +} + +#define VM_AREA_FREE_LIST_MAX 32 + +void vm_area_free(struct vm_area_struct *vma) +{ + struct mm_struct *mm = vma->vm_mm; + bool drain; + + free_anon_vma_name(vma); + + spin_lock(&mm->vma_free_list.lock); + list_add(&vma->vm_free_list, &mm->vma_free_list.head); + mm->vma_free_list.size++; + drain = mm->vma_free_list.size > VM_AREA_FREE_LIST_MAX; + spin_unlock(&mm->vma_free_list.lock); + + if (drain) + drain_free_vmas(mm); +} + +#else /* CONFIG_PER_VMA_LOCK */ + +void drain_free_vmas(struct mm_struct *mm) {} void vm_area_free(struct vm_area_struct *vma) { free_anon_vma_name(vma); -#ifdef CONFIG_PER_VMA_LOCK - call_rcu(&vma->vm_rcu, __vm_area_free); -#else kmem_cache_free(vm_area_cachep, vma); -#endif } +#endif /* CONFIG_PER_VMA_LOCK */ + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -1150,6 +1199,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, INIT_LIST_HEAD(&mm->mmlist); #ifdef CONFIG_PER_VMA_LOCK WRITE_ONCE(mm->mm_lock_seq, 0); + INIT_LIST_HEAD(&mm->vma_free_list.head); + spin_lock_init(&mm->vma_free_list.lock); + mm->vma_free_list.size = 0; #endif mm_pgtables_bytes_init(mm); mm->map_count = 0; diff --git a/mm/init-mm.c b/mm/init-mm.c index 33269314e060..b53d23c2d7a3 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -39,6 +39,9 @@ struct mm_struct init_mm = { .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK .mm_lock_seq = 0, + .vma_free_list.head = LIST_HEAD_INIT(init_mm.vma_free_list.head), + .vma_free_list.lock = __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock), + .vma_free_list.size = 0, #endif .user_ns = &init_user_ns, .cpu_bitmap = CPU_BITS_NONE, diff --git a/mm/mmap.c b/mm/mmap.c index 332af383f7cd..a0d5d3af1d95 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -3159,6 +3159,7 @@ void exit_mmap(struct mm_struct *mm) trace_exit_mmap(mm); __mt_destroy(&mm->mm_mt); mmap_write_unlock(mm); + drain_free_vmas(mm); vm_unacct_memory(nr_accounted); } From patchwork Mon Jan 9 20:53:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094306 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB647C5479D for ; Mon, 9 Jan 2023 20:55:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E979900006; Mon, 9 Jan 2023 15:55:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 79A26900005; Mon, 9 Jan 2023 15:55:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 63B49900006; Mon, 9 Jan 2023 15:55:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 548EF900005 for ; Mon, 9 Jan 2023 15:55:24 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E62D1120C19 for ; Mon, 9 Jan 2023 20:55:23 +0000 (UTC) X-FDA: 80336466126.08.CF7A0DD Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf22.hostedemail.com (Postfix) with ESMTP id 53EC6C0010 for ; Mon, 9 Jan 2023 20:55:22 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sWtobSCB; spf=pass (imf22.hostedemail.com: domain of 3OX-8YwYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3OX-8YwYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297722; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KsiQb/dE9yymRpEr0UBz2Qq4KLrRHehGb+mEU6hVzX0=; b=e9CUsTQYhHEY+CiirFmvbDoAkpsVPiCRe0v7+eKo4Xnfjqg2EVzPFhFgPAWMnEygkl6TgB wue8OPWEzAl0XBQ+m3a3+Liip3AjQd+bG4CU0b9yeh3VxumSZRWwuWRYq8uz+wNq3+LxJL Kz2j7oCgSIrOpyz/e7WvWIA+cmmJAx4= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=sWtobSCB; spf=pass (imf22.hostedemail.com: domain of 3OX-8YwYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3OX-8YwYKCE48A7u3rw44w1u.s421y3AD-220Bqs0.47w@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297722; a=rsa-sha256; cv=none; b=jhygTj+io+Tvo8UBwxJKfcuvoyNKiF8agYaHoOo1IurqiBNgJkDqVbV/xRSqSduaLBxNJf ikk3q7opNc02KTMRHJZWYwmNWCBdf18WfXIw3iYh0/V1jtaxtpSXZYOlg6VXPE3EoePYxj zwKOgtmGA5ojFpDhXAdBCqLgx0NSqgY= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-4bdeb1bbeafso103899627b3.4 for ; Mon, 09 Jan 2023 12:55:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=KsiQb/dE9yymRpEr0UBz2Qq4KLrRHehGb+mEU6hVzX0=; b=sWtobSCBvoHV5TPkE/zYtWszgUW3jVRDiVjTt3sAcjFBaXlqjfdYiQ6AHG86rnNt0b mg+e+VoU+a89JO+rslq34FfyzRkAoGg7DuVDlMH1H1ePQ32MuB2+77t0gAtkJ+y+wX6p B/589/jsqLZApvm6BoQPTAcfoFYkEfWsjxaKMUYIa10AIrZvQul2s59ukdBVjUu/Ibrp egNG28U4wT5FZXec8pRJXGar6bKM6QyHpkRg0GkDbw6/Hl7olgQ7YiTkujC6hmd+0b7u 2fP91n1+O4wNRIiFMGmQ1g/TPPehatod7VMoh0dY0z9ogD+TX+MKDQ5sfW6QhS4fa9cZ O/Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=KsiQb/dE9yymRpEr0UBz2Qq4KLrRHehGb+mEU6hVzX0=; b=qi5gQS6GpYzwrNeDV5mtTrlB3DEZOkQpj0a8C33YdoKJuc1U+3X/C+of24hoCNXBXO o9F2OxZ6Z6/rI9QNOgwPlYfz4dBX+Th2NBRybcCPj6NW4e9kw5luppYgp9+CstS+QABs 8cHkutywuqmoHmi5tZuG0uDe/P5opUY4Sok0+8BNhepX2voF6dPPWn7/r/u2ffm7KApb pFdpacw8wJkHkPpeWIneruOCiZsW70eb9VAytq68J13N6WXQUxty0Jp/9XIS0+WPB9HQ SvVsIo3HZNDvUWssXP9Q4NFjUoICXOVyvVoQYM5qPoK4m5YNrRjeFR+XJcX/yU6xuGLC 5HIQ== X-Gm-Message-State: AFqh2koPh5SZMsxRFokkT4Zr0Inj2+Ci9Guf6pZJXVaxCCHEKNBJmEhU kc/FX8612YkqUlVsAP8IQPbhWBsl6I0= X-Google-Smtp-Source: AMrXdXvB4GUb8flK/fogRdz9aeHP/yrxt0+V7dhaMMs1drQdYkqQuuAWfxuEu+0du5vQqeydWK6w8e3+WKU= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a25:abc6:0:b0:755:29c5:63e with SMTP id v64-20020a25abc6000000b0075529c5063emr6840356ybi.142.1673297721512; Mon, 09 Jan 2023 12:55:21 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:35 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-41-surenb@google.com> Subject: [PATCH 40/41] mm: separate vma->lock from vm_area_struct From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Stat-Signature: yroqndwdujnjttrhcue6ui38rsqnd9dy X-Rspam-User: X-Rspamd-Queue-Id: 53EC6C0010 X-Rspamd-Server: rspam06 X-HE-Tag: 1673297722-835767 X-HE-Meta: U2FsdGVkX1/ktPjlLgrBWPk4010TFxQDAI6kZqxjogieTegcpnIWeoeZoagdTyF8wc7Jf3v1/+s/cH91aLKTdqZYaCmuxAYh4Mv5f1ZOPaGjuHYgFgXOHc7KmiSq2IXmLdYwWQFFH2iVpXX7eTDzVrpKjkwoW+MHZcqWHBw2DApWFOek6Qreu0UHKooug9kFyg0t0Nc27h4ClmSFAphj6UYfqyqgIMsu7vfzHh0C7Lco5f58DMsei0UolbxmD//fsccmnQlECoUpU0vz/bx972XEv/GFPwAfmHiRxdhUuPO5h1IVnSIh246EstYyT9a5tgC3bC/PxYE+xyWwtlkA+GMT1F0+saAc0IPhRY7V+u6OV9jj20y+kOSPT80VIQkxh7X2KBUtKVVJxbv8HzABl2D5Am4i/INxgYpNDghpl8Yvsu4bGuOPMJo8rU4EhuxPbvI5S2uFFgwdum5NR0o1BxUmVEh8ID5X9JHPvYRGjXcmaXXNwAv2baDAdg65PbTlRrx8WnkHqRzXs7J//lq4GOcnMCYsW/MnMU9ghFEBgh+8F3I6bkaxaXLH8gUqRWt3HLIhtG5GD0yQvq+Cu0S4Y9kArCnYeKVcP9+GeqJIFs26gN1RlNqD+wQVxAfFZyK/xCsMExBkkbTRzCqqUHya3zsvwN7RPpnxBAI5kMVr4sa7HE4a6sfXxSS3kRlHJueeN5953k2XL7GRcOl5nIm+AnY/sVVFOntQLZro+MVxzEu77gjJGzYd2Rp5rvhblple3FAKFxmcDj2+iezl0W/wv8m+DiuzWBY1HjRtQfLXXVFv+KX8hSglSG54+b4FIJJGz1QZWJwQYwIQZ3elJdWqeG35JAfZPoM634BpGXK2u/I+sNzcilzba8cutxK0cyn5mvOhwqR88YeATeMTQxEOXUG+QQ8uVy3xpc/F5QBSaBj4uQ0VvhhHPWdxEwOyzE17tote1J8HfBos0zzlSwL LBzlvfLj 3V8BG44vcdiAzYqq3svfiQz8/y8j7ufzZCwvyCdYRc+JIYbc5dLvGc42hmj+gAE+XCQW2ZZM34Qn2TMYEptOip+8iTBJcPcGo53tNVx8XTSUhIMG4rCThY1VpG/rKf7tPDZsSHF5u+shhhj2tNKfq/9jd8iXpYc1YuOj/2CX3juMiuGhav62kpu6AfwzNez/jQSKSJQMvVfmWc+aebmXChGEnhEz8r9BwHcvU9c+PTXyd3tKayp9MR0rB+Z9hxPSaEmrceftfeAVxS57jfU8dEKT5alOOzEfGe2NA0VaOHIwT/k3Tj0+Inuwg55byZNzi78e4czeUHh7iwKHvrb1kwFNdgght9o0Ds26TgBlx1RldUG+3rbiqiK9TI23u07mK801Cw4OOnNoniLWK7KfKGDG6E0eBzj3AcXOE X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vma->lock being part of the vm_area_struct causes performance regression during page faults because during contention its count and owner fields are constantly updated and having other parts of vm_area_struct used during page fault handling next to them causes constant cache line bouncing. Fix that by moving the lock outside of the vm_area_struct. All attempts to keep vma->lock inside vm_area_struct in a separate cache line still produce performance regression especially on NUMA machines. Smallest regression was achieved when lock is placed in the fourth cache line but that bloats vm_area_struct to 256 bytes. Considering performance and memory impact, separate lock looks like the best option. It increases memory footprint of each VMA but that will be addressed in the next patch. Note that after this change vma_init() does not allocate or initialize vma->lock anymore. A number of drivers allocate a pseudo VMA on the stack but they never use the VMA's lock, therefore it does not need to be allocated. The future drivers which might need the VMA lock should use vm_area_alloc()/vm_area_free() to allocate it. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 25 ++++++------ include/linux/mm_types.h | 6 ++- kernel/fork.c | 82 ++++++++++++++++++++++++++++------------ 3 files changed, 74 insertions(+), 39 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 50c7a6dd9c7a..d40bf8a5e19e 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -615,11 +615,6 @@ struct vm_operations_struct { }; #ifdef CONFIG_PER_VMA_LOCK -static inline void vma_init_lock(struct vm_area_struct *vma) -{ - init_rwsem(&vma->lock); - vma->vm_lock_seq = -1; -} static inline void vma_write_lock(struct vm_area_struct *vma) { @@ -635,9 +630,9 @@ static inline void vma_write_lock(struct vm_area_struct *vma) if (vma->vm_lock_seq == mm_lock_seq) return; - down_write(&vma->lock); + down_write(&vma->vm_lock->lock); vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->lock); + up_write(&vma->vm_lock->lock); } /* @@ -651,17 +646,17 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->lock) == 0)) + if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) return false; /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check - * vma->vm_lock_seq under vma->lock protection and mm->mm_lock_seq + * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); return false; } return true; @@ -669,7 +664,7 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) static inline void vma_read_unlock(struct vm_area_struct *vma) { - up_read(&vma->lock); + up_read(&vma->vm_lock->lock); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -684,7 +679,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) static inline void vma_assert_no_reader(struct vm_area_struct *vma) { - VM_BUG_ON_VMA(rwsem_is_locked(&vma->lock) && + VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock) && vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } @@ -694,7 +689,6 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, #else /* CONFIG_PER_VMA_LOCK */ -static inline void vma_init_lock(struct vm_area_struct *vma) {} static inline void vma_write_lock(struct vm_area_struct *vma) {} static inline bool vma_read_trylock(struct vm_area_struct *vma) { return false; } @@ -704,6 +698,10 @@ static inline void vma_assert_no_reader(struct vm_area_struct *vma) {} #endif /* CONFIG_PER_VMA_LOCK */ +/* + * WARNING: vma_init does not initialize vma->vm_lock. + * Use vm_area_alloc()/vm_area_free() if vma needs locking. + */ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -712,7 +710,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); - vma_init_lock(vma); } /* Use when VMA is not part of the VMA tree and needs no locking */ diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index c0e6c8e4700b..faa61b400f9b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -526,6 +526,10 @@ struct anon_vma_name { char name[]; }; +struct vma_lock { + struct rw_semaphore lock; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -563,7 +567,7 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK int vm_lock_seq; - struct rw_semaphore lock; + struct vma_lock *vm_lock; #endif /* diff --git a/kernel/fork.c b/kernel/fork.c index 97f2b751f88d..95db6a521cf1 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -451,40 +451,28 @@ static struct kmem_cache *vm_area_cachep; /* SLAB cache for mm_struct structures (tsk->mm) */ static struct kmem_cache *mm_cachep; -struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) -{ - struct vm_area_struct *vma; +#ifdef CONFIG_PER_VMA_LOCK - vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); - if (vma) - vma_init(vma, mm); - return vma; -} +/* SLAB cache for vm_area_struct.lock */ +static struct kmem_cache *vma_lock_cachep; -struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) +static bool vma_init_lock(struct vm_area_struct *vma) { - struct vm_area_struct *new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); + vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); + if (!vma->vm_lock) + return false; - if (new) { - ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); - ASSERT_EXCLUSIVE_WRITER(orig->vm_file); - /* - * orig->shared.rb may be modified concurrently, but the clone - * will be reinitialized. - */ - *new = data_race(*orig); - INIT_LIST_HEAD(&new->anon_vma_chain); - vma_init_lock(new); - dup_anon_vma_name(orig, new); - } - return new; + init_rwsem(&vma->vm_lock->lock); + vma->vm_lock_seq = -1; + + return true; } -#ifdef CONFIG_PER_VMA_LOCK static inline void __vm_area_free(struct vm_area_struct *vma) { /* The vma should either have no lock holders or be write-locked. */ vma_assert_no_reader(vma); + kmem_cache_free(vma_lock_cachep, vma->vm_lock); kmem_cache_free(vm_area_cachep, vma); } @@ -540,6 +528,7 @@ void vm_area_free(struct vm_area_struct *vma) #else /* CONFIG_PER_VMA_LOCK */ +static bool vma_init_lock(struct vm_area_struct *vma) { return true; } void drain_free_vmas(struct mm_struct *mm) {} void vm_area_free(struct vm_area_struct *vma) @@ -550,6 +539,48 @@ void vm_area_free(struct vm_area_struct *vma) #endif /* CONFIG_PER_VMA_LOCK */ +struct vm_area_struct *vm_area_alloc(struct mm_struct *mm) +{ + struct vm_area_struct *vma; + + vma = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); + if (!vma) + return NULL; + + vma_init(vma, mm); + if (!vma_init_lock(vma)) { + kmem_cache_free(vm_area_cachep, vma); + return NULL; + } + + return vma; +} + +struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) +{ + struct vm_area_struct *new; + + new = kmem_cache_alloc(vm_area_cachep, GFP_KERNEL); + if (!new) + return NULL; + + ASSERT_EXCLUSIVE_WRITER(orig->vm_flags); + ASSERT_EXCLUSIVE_WRITER(orig->vm_file); + /* + * orig->shared.rb may be modified concurrently, but the clone + * will be reinitialized. + */ + *new = data_race(*orig); + if (!vma_init_lock(new)) { + kmem_cache_free(vm_area_cachep, new); + return NULL; + } + INIT_LIST_HEAD(&new->anon_vma_chain); + dup_anon_vma_name(orig, new); + + return new; +} + static void account_kernel_stack(struct task_struct *tsk, int account) { if (IS_ENABLED(CONFIG_VMAP_STACK)) { @@ -3138,6 +3169,9 @@ void __init proc_caches_init(void) NULL); vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT); +#ifdef CONFIG_PER_VMA_LOCK + vma_lock_cachep = KMEM_CACHE(vma_lock, SLAB_PANIC|SLAB_ACCOUNT); +#endif mmap_init(); nsproxy_cache_init(); } From patchwork Mon Jan 9 20:53:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13094307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E564C54EBD for ; Mon, 9 Jan 2023 20:55:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC772900005; Mon, 9 Jan 2023 15:55:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E77FC940007; Mon, 9 Jan 2023 15:55:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1833900007; Mon, 9 Jan 2023 15:55:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BF5C6900005 for ; Mon, 9 Jan 2023 15:55:26 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 83710140C8D for ; Mon, 9 Jan 2023 20:55:26 +0000 (UTC) X-FDA: 80336466252.20.0A44047 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf14.hostedemail.com (Postfix) with ESMTP id D571C10000D for ; Mon, 9 Jan 2023 20:55:24 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=J47RXHly; spf=pass (imf14.hostedemail.com: domain of 3PH-8YwYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3PH-8YwYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673297724; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6eLzL6RHeFdZbRB5oFC0iJ795TBzqFT5ovI/MwJrKVk=; b=uvaskqy8h1cXYBjdh2x1thYjY1bDmm+MMjz96hjkX3dEHX8BRHiuiuIYqms39iskxyW/6U +KbOnbNnUSUSiFfPh/kEsezR88W4SPkXKxzm07vFTyq6gcUzFlk7Rj09VOMA74Jw8OHdRW V8imsyLBMnSUcC7W1ZVTnUdpywIwjgc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=J47RXHly; spf=pass (imf14.hostedemail.com: domain of 3PH-8YwYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3PH-8YwYKCFEBDAx6uz77z4x.v75416DG-553Etv3.7Az@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673297724; a=rsa-sha256; cv=none; b=z6NZzoDQ/HRSRK+D5NrqRrpPbMpQvCVRg8coRymaOETsq+6aYVefn2JkMrIAOuqm/4vKtK jvdzUhgKSXT8mPCEXp5Gn/rCjBmPouILjzYbgFQ/3WT8QdVECDhbYNM9QMbT6IlxpQVo82 vjksbQg3X4wMtYAU9VG+8jz9nYQqIP4= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-487b0bf1117so104173037b3.5 for ; Mon, 09 Jan 2023 12:55:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6eLzL6RHeFdZbRB5oFC0iJ795TBzqFT5ovI/MwJrKVk=; b=J47RXHly65L6DdSqhS16NmYDveuNz+mvTNG4sGvbQhUoWkeX6DBvsEbauZWeZvA/FL 6iJWwlnZTOJwBfCXcZD78WFVJCG7QdSQyabstEWgAbqGHA3DDp9ZOq1VOAfH1PKWr803 hD+hbwJSt9HgzHQXXidFA8sLeLstkKuaZuBnozg86r7c8tgk1iKjkQ1PqFU36BM8+uUF XrYI9BSRx6s/UM8T/tSpiSH9gONtyaFYLkfVKHbhv2aDCwIEnthDk+4faf5N9RzawpoY Qe27f6FOB5GsQ/NQQedhE4ZJmkp3i9Bbttmr9VE8Pj62jtmxe5G6ZqBn6CUeB13gsJFT GliQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6eLzL6RHeFdZbRB5oFC0iJ795TBzqFT5ovI/MwJrKVk=; b=LDkIWZe9OT4H2j1VidEElfNYxj4JiXJiDPD+BoXgHsr+xsUCCwQf4bGFyoIG1C8Hua 9vg6CRRoBdbamLE8Z2wN30rx/MHEN5VmCO9HYZlXIG2Wihtymv54p6Q6twzDRnR9Hren KCuscQSIY4tHsKUBD+7Nk/CrTwDysHvDeQ7LbCsLuXoFlGYkpfQHJ6a4GEh65Y/SQTnn zxAgYtRL8UjfUFMKxHbnJhPCgaQ3fMEal0USjtozChM932sjXVk8Ob1XxzgmeVnMvI1V F0g/pBtC8ZzCeOkhiZo8GDI3Yapg0BAllpAlYqm/eqDapmSWKCKzr4Qmvn7gU6ki6VOi +28g== X-Gm-Message-State: AFqh2kqh6d1JXyGwSA40krGjkAKHZAEYvDcSpHuMhK7ZpD0FEWp9yXfx /fSj2VfWJWItmta16H2t9qY4PctYDvo= X-Google-Smtp-Source: AMrXdXvrHv3ZAwS3fQ/OCRP/gup9EskfAqpVKQwrbofhFp/7IXGRuSpH+yzdGI9HlCLagGnSix+y0aiJr6A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:200:9393:6f7a:d410:55ca]) (user=surenb job=sendgmr) by 2002:a5b:305:0:b0:707:473f:b763 with SMTP id j5-20020a5b0305000000b00707473fb763mr7071095ybp.158.1673297724036; Mon, 09 Jan 2023 12:55:24 -0800 (PST) Date: Mon, 9 Jan 2023 12:53:36 -0800 In-Reply-To: <20230109205336.3665937-1-surenb@google.com> Mime-Version: 1.0 References: <20230109205336.3665937-1-surenb@google.com> X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230109205336.3665937-42-surenb@google.com> Subject: [PATCH 41/41] mm: replace rw_semaphore with atomic_t in vma_lock From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: michel@lespinasse.org, jglisse@google.com, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, mgorman@techsingularity.net, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, peterz@infradead.org, ldufour@linux.ibm.com, laurent.dufour@fr.ibm.com, paulmck@kernel.org, luto@kernel.org, songliubraving@fb.com, peterx@redhat.com, david@redhat.com, dhowells@redhat.com, hughd@google.com, bigeasy@linutronix.de, kent.overstreet@linux.dev, punit.agrawal@bytedance.com, lstoakes@gmail.com, peterjung1337@gmail.com, rientjes@google.com, axelrasmussen@google.com, joelaf@google.com, minchan@google.com, jannh@google.com, shakeelb@google.com, tatashin@google.com, edumazet@google.com, gthelen@google.com, gurua@google.com, arjunroy@google.com, soheil@google.com, hughlynch@google.com, leewalsh@google.com, posk@google.com, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, x86@kernel.org, linux-kernel@vger.kernel.org, kernel-team@android.com, surenb@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D571C10000D X-Stat-Signature: 97ts5zz9jksfwhu68u6aegrbg4e6yybq X-HE-Tag: 1673297724-429836 X-HE-Meta: U2FsdGVkX1+5AS7VbUSrPft/Knw6ir90W4jYcSLWGax7IUyJR2n39JF4/u0Hh49lPaXGWWt0CSTOdlAUYiKIWcXR0kZoOfCFd94+XzbCb+aFZqFN8GsvTATrHk7jf05U9vePn6ksAwp9TOFpWLJRspmb9ak/T5fZBs8JhsFxnkn2VGFLCQJEzlVNvKcchCsgJ4n2TImtybvheWGYmvsDs8CbpC2JLz0hcvgwBdo36XyKp/dMDDxNeYwIjvWNETfxyqsHXMFkVkoLnHCKomnVBWIqTiCJ31XW0RjugomQTN0e/sQG7w10KvhBoV8hUI2D24x1+ZqqQHJnDzAFfyKSSlEL7RQVNUIMrhUkx8Nw9DD/CsDcSrXv2RPP2L/w5QfMUvAzDylTl4t6S62PqQsWc8ZRIZsvsiZYtKnZ11vvd2QIEOmwbdCEQl9Ap+GithhPRqww0H5cB9EjZaeFauUtIdjtP7Ei26hxuqH86nPdQAh1hCU3FmG9Rwh8o0k3ij4Mzv0LxWDv7NLnt8nEfpTvc5Dnja8v3P7ADgv69p9z/ZGUTOl0ho5AlToyFJILz3Ga+kSIe/6uA6PfUOAwhSErV8Og1nRWlg6rN85C3fxiMRD9aQVYOoDWviUs3xiQ+93Adu8YgC1pB+zxv1P8fMrPsohMQHBYBWDAjD2ROpXgoKkx/Pt94wKeQ/tNB8glaEgGnumo72WZB1YO7NGI176WfVJsL4UCS6BNxkjTEIr69Jc+2MVLF8VxxF54th4nY9lO4pkE9achg1BE2wjiMjUK13N8l3Jsoa6klkSrMhs8AG9aOUb/DqTOCKi7q++phi1JjmtsaacflHmcIz3f6/wNZrh1HRXEYax4sS//fMO/fI0WwYAU3nRZ9UI09IWnxcf7pv2J4fpLe4xPYfiKEqedVCg0SPSugvoLjb+B6XbG0SGffjT75tDc7WCy+v176t5WI8ePUOqQvg7VGCtxyaM tsv19k5o lntTDxsK9Pi5mmkXGNDuQYusaycc6/peunrVFQ+eFk7L9qD1FzT5Jbs/SbBwv/x+GKTUX8PJ43s5FCNqEoYbl4EUKstf9nbwekuFydFR4bcNiw+FNuFZgQ+e9hizJ/Npm49XvJuIB5wOnigKm9X5iFxGHCFuZ4uIW/ffwLa00vZdz0P43eL+vyu8NyHqs2fXE3w0CZzNNJNFnYb4odkR0M3DLw1z1YJAPANo0GtIIUI+TcvmSNrNa4Mo+A39QWZoZnQtxV/9HVr93KSZVVDjNSGgNO7xME1W1boK10VrdK3Lg+DdrzFwoOG1sMIGP6jCdpYMOlbHelfRp79aWeO07yfJgtnVHMsep8cuYThkNfU5s7m/j4iS91EBVkN5QIxEiTQw8jqR5kk9HAyfKPQCnJ9Gl3a9CXodjnJwG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: rw_semaphore is a sizable structure of 40 bytes and consumes considerable space for each vm_area_struct. However vma_lock has two important specifics which can be used to replace rw_semaphore with a simpler structure: 1. Readers never wait. They try to take the vma_lock and fall back to mmap_lock if that fails. 2. Only one writer at a time will ever try to write-lock a vma_lock because writers first take mmap_lock in write mode. Because of these requirements, full rw_semaphore functionality is not needed and we can replace rw_semaphore with an atomic variable. When a reader takes read lock, it increments the atomic unless the value is negative. If that fails read-locking is aborted and mmap_lock is used instead. When writer takes write lock, it resets atomic value to -1 if the current value is 0 (no readers). Since all writers take mmap_lock in write mode first, there can be only one writer at a time. If there are readers, writer will place itself into a wait queue using new mm_struct.vma_writer_wait waitqueue head. The last reader to release the vma_lock will signal the writer to wake up. vm_lock_seq is also moved into vma_lock and along with atomic_t they are nicely packed and consume 8 bytes, bringing the overhead from vma_lock from 44 to 16 bytes: slabinfo before the changes: ... : ... vm_area_struct ... 152 53 2 : ... slabinfo with vma_lock: ... : ... rw_semaphore ... 8 512 1 : ... vm_area_struct ... 160 51 2 : ... Assuming 40000 vm_area_structs, memory consumption would be: baseline: 6040kB vma_lock (vm_area_structs+vma_lock): 6280kB+316kB=6596kB Total increase: 556kB atomic_t might overflow if there are many competing readers, therefore vma_read_trylock() implements an overflow check and if that occurs it restors the previous value and exits with a failure to lock. Signed-off-by: Suren Baghdasaryan --- include/linux/mm.h | 37 +++++++++++++++++++++++++------------ include/linux/mm_types.h | 10 ++++++++-- kernel/fork.c | 6 +++--- mm/init-mm.c | 2 ++ 4 files changed, 38 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d40bf8a5e19e..294dd44b2198 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -627,12 +627,16 @@ static inline void vma_write_lock(struct vm_area_struct *vma) * mm->mm_lock_seq can't be concurrently modified. */ mm_lock_seq = READ_ONCE(vma->vm_mm->mm_lock_seq); - if (vma->vm_lock_seq == mm_lock_seq) + if (vma->vm_lock->lock_seq == mm_lock_seq) return; - down_write(&vma->vm_lock->lock); - vma->vm_lock_seq = mm_lock_seq; - up_write(&vma->vm_lock->lock); + if (atomic_cmpxchg(&vma->vm_lock->count, 0, -1)) + wait_event(vma->vm_mm->vma_writer_wait, + atomic_cmpxchg(&vma->vm_lock->count, 0, -1) == 0); + vma->vm_lock->lock_seq = mm_lock_seq; + /* Write barrier to ensure lock_seq change is visible before count */ + smp_wmb(); + atomic_set(&vma->vm_lock->count, 0); } /* @@ -643,20 +647,28 @@ static inline void vma_write_lock(struct vm_area_struct *vma) static inline bool vma_read_trylock(struct vm_area_struct *vma) { /* Check before locking. A race might cause false locked result. */ - if (vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) + if (vma->vm_lock->lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq)) return false; - if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0)) + if (unlikely(!atomic_inc_unless_negative(&vma->vm_lock->count))) return false; + /* If atomic_t overflows, restore and fail to lock. */ + if (unlikely(atomic_read(&vma->vm_lock->count) < 0)) { + if (atomic_dec_and_test(&vma->vm_lock->count)) + wake_up(&vma->vm_mm->vma_writer_wait); + return false; + } + /* * Overflow might produce false locked result. * False unlocked result is impossible because we modify and check * vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq * modification invalidates all existing locks. */ - if (unlikely(vma->vm_lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { - up_read(&vma->vm_lock->lock); + if (unlikely(vma->vm_lock->lock_seq == READ_ONCE(vma->vm_mm->mm_lock_seq))) { + if (atomic_dec_and_test(&vma->vm_lock->count)) + wake_up(&vma->vm_mm->vma_writer_wait); return false; } return true; @@ -664,7 +676,8 @@ static inline bool vma_read_trylock(struct vm_area_struct *vma) static inline void vma_read_unlock(struct vm_area_struct *vma) { - up_read(&vma->vm_lock->lock); + if (atomic_dec_and_test(&vma->vm_lock->count)) + wake_up(&vma->vm_mm->vma_writer_wait); } static inline void vma_assert_write_locked(struct vm_area_struct *vma) @@ -674,13 +687,13 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma) * current task is holding mmap_write_lock, both vma->vm_lock_seq and * mm->mm_lock_seq can't be concurrently modified. */ - VM_BUG_ON_VMA(vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); + VM_BUG_ON_VMA(vma->vm_lock->lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } static inline void vma_assert_no_reader(struct vm_area_struct *vma) { - VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock->lock) && - vma->vm_lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), + VM_BUG_ON_VMA(atomic_read(&vma->vm_lock->count) > 0 && + vma->vm_lock->lock_seq != READ_ONCE(vma->vm_mm->mm_lock_seq), vma); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index faa61b400f9b..a6050c38ca2e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -527,7 +527,13 @@ struct anon_vma_name { }; struct vma_lock { - struct rw_semaphore lock; + /* + * count > 0 ==> read-locked with 'count' number of readers + * count < 0 ==> write-locked + * count = 0 ==> unlocked + */ + atomic_t count; + int lock_seq; }; /* @@ -566,7 +572,6 @@ struct vm_area_struct { unsigned long vm_flags; #ifdef CONFIG_PER_VMA_LOCK - int vm_lock_seq; struct vma_lock *vm_lock; #endif @@ -706,6 +711,7 @@ struct mm_struct { * by mmlist_lock */ #ifdef CONFIG_PER_VMA_LOCK + struct wait_queue_head vma_writer_wait; int mm_lock_seq; struct { struct list_head head; diff --git a/kernel/fork.c b/kernel/fork.c index 95db6a521cf1..b221ad182d98 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -461,9 +461,8 @@ static bool vma_init_lock(struct vm_area_struct *vma) vma->vm_lock = kmem_cache_alloc(vma_lock_cachep, GFP_KERNEL); if (!vma->vm_lock) return false; - - init_rwsem(&vma->vm_lock->lock); - vma->vm_lock_seq = -1; + atomic_set(&vma->vm_lock->count, 0); + vma->vm_lock->lock_seq = -1; return true; } @@ -1229,6 +1228,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, mmap_init_lock(mm); INIT_LIST_HEAD(&mm->mmlist); #ifdef CONFIG_PER_VMA_LOCK + init_waitqueue_head(&mm->vma_writer_wait); WRITE_ONCE(mm->mm_lock_seq, 0); INIT_LIST_HEAD(&mm->vma_free_list.head); spin_lock_init(&mm->vma_free_list.lock); diff --git a/mm/init-mm.c b/mm/init-mm.c index b53d23c2d7a3..0088e31e5f7e 100644 --- a/mm/init-mm.c +++ b/mm/init-mm.c @@ -38,6 +38,8 @@ struct mm_struct init_mm = { .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist = LIST_HEAD_INIT(init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK + .vma_writer_wait = + __WAIT_QUEUE_HEAD_INITIALIZER(init_mm.vma_writer_wait), .mm_lock_seq = 0, .vma_free_list.head = LIST_HEAD_INIT(init_mm.vma_free_list.head), .vma_free_list.lock = __SPIN_LOCK_UNLOCKED(init_mm.vma_free_list.lock),