From patchwork Thu Dec 28 08:46:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13505703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2FB5C3DA6E for ; Thu, 28 Dec 2023 08:59:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4075E6B00A2; Thu, 28 Dec 2023 03:59:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3B7A66B00A3; Thu, 28 Dec 2023 03:59:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27F8E6B00A4; Thu, 28 Dec 2023 03:59:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 189386B00A2 for ; Thu, 28 Dec 2023 03:59:32 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DE2B3C0816 for ; Thu, 28 Dec 2023 08:59:31 +0000 (UTC) X-FDA: 81615628542.07.6A0E736 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf04.hostedemail.com (Postfix) with ESMTP id 18A3340014 for ; Thu, 28 Dec 2023 08:59:28 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="VPt/k23K"; spf=pass (imf04.hostedemail.com: domain of jszhang@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=jszhang@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703753969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rbu3zDK4egabfYRNLq5ZDg2lMGMbWX/VoBKV5Lc26JI=; b=ntpS0Em+CGrLuxsaMBbesy2WeFqE+7iiqCORbnX/xK50sKazLNG474/Wmcib03KcKDScC+ ohF4dgZSZ24xTYzQ+BLlKM4PyHvS0PGHG7/zhUcJDjO0Nd8CmQQ4iQUsgk8IzP/x9DAHuI 0yAQL1TH1y5SZ7hSjdnUbuvygTtrIgU= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="VPt/k23K"; spf=pass (imf04.hostedemail.com: domain of jszhang@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=jszhang@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703753969; a=rsa-sha256; cv=none; b=QQLGm6DUXlZWV9CKXF10qbl4KvcAtfi9o2HJkrMioNmi83Q5qhvJ8D1Ecsn8NAOs9LJIm+ +qHd1J6cIT8YZR5IWRCLBYUtXfx+hBm4vBAAVdi146gQTkKwpd8KwBZNjHj7wIk4Iw0Ucr ajB+FKMRGnjqaRNGpB7jF76sx1NDAY0= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by ams.source.kernel.org (Postfix) with ESMTP id 89025B80D97; Thu, 28 Dec 2023 08:59:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB820C433CA; Thu, 28 Dec 2023 08:59:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703753966; bh=16+Nu0Wb1JkSQStq2mcU9kdq58a4r2QX4o0XnGMbW1g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VPt/k23KQR/ek+Wk4Mb0Ffgqjpr0C4tGoF8uMnFE+csMqgfr093NXmCFQN67kLxmg 2wX671IgLVNFprhYGlOqZT9sQd45M//tMA96Y48PuyxuAUw2YYrGJheIf3jUNzegsO oZr1LwJ+Pd98LSUtkZQ1GQJGY/kKAzRUqhRAQ2LGss8txZ4sD1RLgnpeb+dxBj+bEm lTFy+2WWHg42Kl6nbbR4U7U8bNB7+1AF95XqLf72vf4OQxZI1A08+R/V8QaP95mAv4 W39UhXGY1i/064M112ShThexcH0ieACEEyC3qc6Oesf9LT7f5gDAuaHeHCYJG86mq+ AIf2i7+FdSkcQ== From: Jisheng Zhang To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Catalin Marinas , Paul Walmsley , Palmer Dabbelt , Albert Ou , Arnd Bergmann Cc: linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Nadav Amit , Andrea Arcangeli , Andy Lutomirski , Dave Hansen , Thomas Gleixner , Yu Zhao , x86@kernel.org Subject: [PATCH 1/2] mm/tlb: fix fullmm semantics Date: Thu, 28 Dec 2023 16:46:41 +0800 Message-Id: <20231228084642.1765-2-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231228084642.1765-1-jszhang@kernel.org> References: <20231228084642.1765-1-jszhang@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 18A3340014 X-Rspam-User: X-Stat-Signature: 65rcaxzzopkg7fr4httxs81dmdj8biuu X-Rspamd-Server: rspam01 X-HE-Tag: 1703753968-457635 X-HE-Meta: U2FsdGVkX1/qj/TomgDKQbVtKlJ8ulwRxlXNBK1qjyV5j8961KLZxSi23e9UF0XFTZHo9Xg2hIfO9H+2/UprxJXZZ1PUy+baMACDj9xvA8Vt1o9mPkm7MnSiSiiVUZY5XIjpfBNqdbmX7d5kpPgD5yLoD/YUkQbvbV9PCnzF9xmw+osbYMvnc6vZg4tAjmptSuzzYsWLhHL5kPTMHrTNh4WEDkE95ZZa01ZnKYSneFOvGJSvX3AdoZSpakqyhitadC1uRDpZrKotw9LJ7c0BS6Y46xHjopxqftj5le49jDyUz4eFR/YYupLUHe/4HNXEy+pYm47i3foAog3wlbOtaCh6t7JJX0WYvM13cDcthfvI+nluNkQ/Ql0/pn53xxDijiLHUBFEky4UrszowiAH1v9bjUpqA5Y+sCkD+aosCS5llPm2U8IeVTw3Nkb8xNFg/zO7gamfbHv/Kp8amcECHJlVewqgSqlVM0hgNB0ixbatI5aavPK9ItpBNQVAu5CpBzrGex7OaVF1nCbj8JcctUukrfi3M0X2wRkmt5WegyATUtJZpcNmnQ1BjVqhW2HvQonC/GV0EUk/A5pTaYyA7F8Zw2194MY7ftl+avDA5Wexge0mf1Y+YNwuvGYYuXu0hEPDokaZygcTnuA7JWEGVx2TIFWJ9/3oUu2n0rqDWJZAPpjWs0FivfYfgJpCYHoETEU9Suv4WnrtGGxvnYBintQF6DQbFw/tkyovrnZ2gi0RIODSjLEx6Djd/apP4hzoGDNq9ecIo+Ql3qKttailv8Y5Fu9fQzH3nSIBXMpY4RqJsRlZOwmBv1VJCj70b7SNn3e9xcfs63AxWm2P2B14TLrU0q52PUf8767O9LmaHqjcNDRQWDWR2Ixt2MK+2eM0b4j8mR6BGnQ7Dg7BJwottO4hl4p75a3k8ERfxRq7T55XRm7ODMJQ/cHzzcZ7+kIfY6pR00lfxu9xsspXV3m fyf8fjai fxzfANJkp6MpcuUkI0rPzuDBh/oCfGxyIb3owZBbZKMoa3N2UgjuVNp5F6ahd9vMBzVLFT6ZmoEspAe3dTiW8In6rB1go+HjMmbESimDs9eVWt17ZeyUrg3lmgtAvMPM/6NVtta0lTAcyk2MJeifwDMvPaTC56av4afABRyxtGfNdzlXpzzj6RBHM/+J+PI1Crn5lg5TRHX3es3dGbtGSoE1yXeaKXQx7URm3KKNj6T2YJdJBMZ1XaKcoURuRgP8RSLLSK0/2hAkjhLA4YX5IEV6VMB4pgZmGuctUDmSXwG9ml4UfkYbXeTOg5lto+pdt/zVovGBt7lF1mwBY9xAMiwSN00bXxAlxok9mm8Tilj4/Ji1yIX2uWBRiMvuC6jx425EM2wjvkeltjg0jYXUeCLNfLIXwQ+EF/sMYm6g7nTBu5dlha+64X1Kcw8rv3mQjyGjszZJCz218UPovC+SIj7eVXBV+vXsBtpnD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Nadav Amit fullmm in mmu_gather is supposed to indicate that the mm is torn-down (e.g., on process exit) and can therefore allow certain optimizations. However, tlb_finish_mmu() sets fullmm, when in fact it want to say that the TLB should be fully flushed. Change tlb_finish_mmu() to set need_flush_all and check this flag in tlb_flush_mmu_tlbonly() when deciding whether a flush is needed. At the same time, bring the arm64 fullmm on process exit optimization back. Signed-off-by: Nadav Amit Signed-off-by: Jisheng Zhang Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Cc: x86@kernel.org --- arch/arm64/include/asm/tlb.h | 5 ++++- include/asm-generic/tlb.h | 2 +- mm/mmu_gather.c | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 846c563689a8..6164c5f3b78f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -62,7 +62,10 @@ static inline void tlb_flush(struct mmu_gather *tlb) * invalidating the walk-cache, since the ASID allocator won't * reallocate our ASID without invalidating the entire TLB. */ - if (tlb->fullmm) { + if (tlb->fullmm) + return; + + if (tlb->need_flush_all) { if (!last_level) flush_tlb_mm(tlb->mm); return; diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 129a3a759976..f2d46357bcbb 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -452,7 +452,7 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) * these bits. */ if (!(tlb->freed_tables || tlb->cleared_ptes || tlb->cleared_pmds || - tlb->cleared_puds || tlb->cleared_p4ds)) + tlb->cleared_puds || tlb->cleared_p4ds || tlb->need_flush_all)) return; tlb_flush(tlb); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 4f559f4ddd21..79298bac3481 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -384,7 +384,7 @@ void tlb_finish_mmu(struct mmu_gather *tlb) * On x86 non-fullmm doesn't yield significant difference * against fullmm. */ - tlb->fullmm = 1; + tlb->need_flush_all = 1; __tlb_reset_range(tlb); tlb->freed_tables = 1; } From patchwork Thu Dec 28 08:46:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13505704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15254C4706F for ; Thu, 28 Dec 2023 08:59:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A4EC6B00A4; Thu, 28 Dec 2023 03:59:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4304C6B00A6; Thu, 28 Dec 2023 03:59:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31C876B00A5; Thu, 28 Dec 2023 03:59:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1B81C6B00A3 for ; Thu, 28 Dec 2023 03:59:33 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F0566160844 for ; Thu, 28 Dec 2023 08:59:32 +0000 (UTC) X-FDA: 81615628584.03.DE06240 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf04.hostedemail.com (Postfix) with ESMTP id 6A39D40008 for ; Thu, 28 Dec 2023 08:59:31 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TKBsmvO8; spf=pass (imf04.hostedemail.com: domain of jszhang@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jszhang@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703753971; a=rsa-sha256; cv=none; b=EMEmJRWmA/LWTxA5HX4eTnBKpdP6sE/r5sKuTq9Xv4zZk53iFVM9cGzmgXitbYTvvlY9rC /EruhoL7tEI58bgTBTVShZX0XSCgbK9gbFYEGNvs+trHdcGDXMu6qkWu9G1GGF8PHjAkDu HFeQlvjK02PvIOY0TFnRZaAgPO157dw= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=TKBsmvO8; spf=pass (imf04.hostedemail.com: domain of jszhang@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jszhang@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703753971; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U0bHBCfybk8M/ty/5LuxJpVChy9zxALoGJGRJoUUtYY=; b=dpk1zfY8/XtpxsFzWXTUt/9TB+bU5jDyGQpSk1yXyFclZ2zAfVPKsHbHF3ai6qLVR5Zj/t 5eEjBcP1tii6PubEabktUSKeEZ5lUWvBk2JmlFr25K1nsnGyaDR2xsdrn5Q9JpvQUqdjDp ELXWOc+qs+b7rpMiGnF8BCyzYFPtutU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 4F39E60FE6; Thu, 28 Dec 2023 08:59:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5389DC433CC; Thu, 28 Dec 2023 08:59:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703753970; bh=4x9t1f04s06EEvjvtQ7yC2/ljPKJqNDBrcj2L/hT3Js=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TKBsmvO8700bopUsWROGXLmoCWFXsTW+dDVnNGSxOkaL/USi53RKl8TdsixW0+meq ZCWxQdbp33ikg/G9ezCtq7NkctrZnj90/OzQ0n9iUxG+YKDW4DQ0nPcXFY42wiBv+r 997dRhgKccetlwPkyjqAZY+f5mx70kEYFtXj+CoiB8mr5EJbZvQA7ejpm/TS62Kx8Z MVCzbTzHlBLytfAxGrORGDpTnrYATlbt8PPwaCWGIg2WfXt/VlliMYXrTExd8VnHSR LxHkWdaNYJGBiXawHPjLTvQ6a2cSRLkXc9Y/wtpnv3MN3QHCLVBpveSUN4P0ShLMkB BBgaja1UR4WBA== From: Jisheng Zhang To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Catalin Marinas , Paul Walmsley , Palmer Dabbelt , Albert Ou , Arnd Bergmann Cc: linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH 2/2] riscv: tlb: avoid tlb flushing if fullmm == 1 Date: Thu, 28 Dec 2023 16:46:42 +0800 Message-Id: <20231228084642.1765-3-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231228084642.1765-1-jszhang@kernel.org> References: <20231228084642.1765-1-jszhang@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 6A39D40008 X-Stat-Signature: op746j165zrwdq8bqaaijx4x15c7qmwb X-Rspam-User: X-HE-Tag: 1703753971-836905 X-HE-Meta: U2FsdGVkX1/TgNKB3XuOUezxiL8a79hCivQHeVVYBZ6sLfHtLYmsJfCKdi1jq6vHNIF98NVord2Tx9iOoFnCGkmSFgZIYkC+ZUZW7IzKwS0QJEDd4wboD/i61jipGd6IhlgVsOVJsqGAh7Rzso0jRCAZCZbczAHxz6QE6OjkTfyxTGgv0gMueUbVwjpe+zUpDRsLBMjPk+QT1s+mFHpdF1ufhoL0Igb6PBy9BNYtAsu3yhCIajVqoBf9/vsT6RJNmX4rgb28VtX5Nh8VSzLMIZ6us0lSmurYx6JwWbVPPpWvob6sIR5ScNF3VzLhZgsm3WG/yZm8odJBe4ZMYQBhwPUMfLj5KDKGo2r/EdQjM4196dP7KXXO18or6B9kCDuzqt4Bv+pnFSa4XOMaFSPsTbhbjG/umjSmTSkvu/JbBsN7mc/kaACehOjlu0J/kOEJlPQbMVRT03R8k8tZpnmxaZF8h6yC78XU2XvXv55XaP+9Q2sbrii5x39oft8i901PSTgLK8sQLfKUJlU9DSkFvM+zXu2rUiMvS0xV2ovlt61szkh3mtlKTX0nC8g+eygcvzV6BW+hNetXhvBF/3zxbEkwPIMYdbQNc/7B1b1Pa/InH6gyTK1ixa0ocoR6nSccEHZ+hzLkLDvvOIcigUNAALahPeWfOYSE66TUAwv3Iivjbk72Dwfi1GP/5kZ/vv8JwORf3uw4VSZE99d7klXhZtzvvTdNH6DZXq+yS14IIyP4zYJAya6NgplggwyB3eUA2fPNK15P/ZUx22WYTKOxgQPQMNTZfbX4ngD4c3/pEe5KeYq9OG7ndyVZTj3aamrA0IrDFWYbkWVhaXQQIW4LiRv/4O1ykBMxBtISefZ9ynuwb/MTOVYgmVlY0GUJj7H59NkDSwVRe/atnxKVg7BUl28pFAgwVi6vVRT5XYcdpxYs2bu4iXfIfzyPjzuRXr/QwjjCzrbeBQR1dBMAmNc D7ssb6AG UiXRzYkLTv3EVsXbktPV1B8MCnQoixdNmeXNVf6pMLHXzsnhduv4kLwukDA2hnjzjqGsWAdur5pYbFebevR+H4vJy9js8TRSHTczKA7WNUz1OZh9/JbqRdMQpxBsEUjwGQIcsmoue1VQ8lB9Dhyh+bg3mJ1SGG2uoWrzpmGGyymsblJel/m51fV/gWuJGtektI0YiiZHhy8bK9fpBJYMX86GPDodqjCNqR16ovV+CgSMx7bl8G2kZY4wipw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The mmu_gather code sets fullmm=1 when tearing down the entire address space for an mm_struct on exit or execve. So if the underlying platform supports ASID, the tlb flushing can be avoided because the ASID allocator will never re-allocate a dirty ASID. Use the performance of Process creation in unixbench on T-HEAD TH1520 platform is improved by about 4%. Signed-off-by: Jisheng Zhang Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/tlb.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1eb5682b2af6..35f3c214332e 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -12,10 +12,19 @@ static void tlb_flush(struct mmu_gather *tlb); #define tlb_flush tlb_flush #include +#include static inline void tlb_flush(struct mmu_gather *tlb) { #ifdef CONFIG_MMU + /* + * If ASID is supported, the ASID allocator will either invalidate the + * ASID or mark it as used. So we can avoid TLB invalidation when + * pulling down a full mm. + */ + if (static_branch_likely(&use_asid_allocator) && tlb->fullmm) + return; + if (tlb->fullmm || tlb->need_flush_all) flush_tlb_mm(tlb->mm); else