From patchwork Thu Jul 7 12:52:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 12909534 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C87FC433EF for ; Thu, 7 Jul 2022 12:53:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 870896B0073; Thu, 7 Jul 2022 08:53:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 820526B0074; Thu, 7 Jul 2022 08:53:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E80D6B0075; Thu, 7 Jul 2022 08:53:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 625FF6B0073 for ; Thu, 7 Jul 2022 08:53:12 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 3412F80E19 for ; Thu, 7 Jul 2022 12:53:12 +0000 (UTC) X-FDA: 79660294224.29.20DB1D8 Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf22.hostedemail.com (Postfix) with ESMTP id CACEFC0057 for ; Thu, 7 Jul 2022 12:53:11 +0000 (UTC) Received: by mail-pg1-f175.google.com with SMTP id r22so11440965pgr.2 for ; Thu, 07 Jul 2022 05:53:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OkXjFR+NYH0OoqXK6diEul5vRaNtuMp4IaWe2/sFDXI=; b=nuqQ+wJ5o95y7jGGqpjCELTfPjPUHgi7E7FTZs0gaGOCcqFLscKyTXuknhl2r0Tr8i GOKY+oU6XKhO8aGgFKaqc5b+9pKcLFKJ7RiiUd+QY6xRysPA/boO7vrt9gotnDM1k9VM zIFmA8SksMiiE9HHckgX6o3SgQiX8ApGKfCA6UHTrz2tEszAPlMUEjm835NOJBg9WeDe CzWBlbfIOMzbr9uTof81U+ft8J7xcwV9GCdQefl3HC72cHN8J4/cN0Wni9gPCz5OCWIl olZjBjGuiap0t573baIhDwmeLQibM5GcnlMYqdobD9no36jFTc8y4Ce7Val9hIgEyvo+ VDXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OkXjFR+NYH0OoqXK6diEul5vRaNtuMp4IaWe2/sFDXI=; b=hF5LjYWJaFJhV8292rDiWcHDDhgPtgHQp9Oyo201jJC8oo+E35QfYf1KUvRjr1K5Y9 Dtig0z8eJaH8/CXQs5wVhZr3XeLwGFgYY2d+deV4GL/A6p60fuiVNnvCNS/hiRnpGdvm ZpPdvvOcW0lthQadMQt7C4GwuTzbhibHX1RVXrshltq9scS119KsuzGeSVxSgaVjT51u hpx30ogjWvV3b2jYTt8fe3KHXhrlmPJRsHtR9zyaw/LrRN+dcrobfU0QuE8iP5t7O1oW 9Yg8pEsE+zBf9+XjvZBId6iTjH9EUGbQGnKPAYhhHDqu9Km0DtX5eZca0zSyR2rN6A00 Sf/w== X-Gm-Message-State: AJIora8wfxHOU5DDMBhmy2l2/RkCaQp+OSoCAgOBr1+itKw98gnQvynW ZxLrsevRqCVgEpt/CTws83k= X-Google-Smtp-Source: AGRyM1v1964MD/IIcSfx/8Y5slyYPx4KMBqa/W04tHsO/yrwZoWnhZV8fVssXI0+aMYp8g1qivWNHw== X-Received: by 2002:a63:f415:0:b0:408:808b:238f with SMTP id g21-20020a63f415000000b00408808b238fmr38849185pgi.469.1657198390801; Thu, 07 Jul 2022 05:53:10 -0700 (PDT) Received: from localhost.localdomain (47-72-206-164.dsl.dyn.ihug.co.nz. [47.72.206.164]) by smtp.gmail.com with ESMTPSA id a11-20020a1709027e4b00b0016a3b5a46f0sm27907831pln.241.2022.07.07.05.53.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Jul 2022 05:53:10 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org Cc: corbet@lwn.net, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, yangyicong@hisilicon.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, Barry Song , Nadav Amit , Mel Gorman Subject: [PATCH 1/4] Revert "Documentation/features: mark BATCHED_UNMAP_TLB_FLUSH doesn't apply to ARM64" Date: Fri, 8 Jul 2022 00:52:39 +1200 Message-Id: <20220707125242.425242-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220707125242.425242-1-21cnbao@gmail.com> References: <20220707125242.425242-1-21cnbao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657198391; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OkXjFR+NYH0OoqXK6diEul5vRaNtuMp4IaWe2/sFDXI=; b=vHwS8ZVFTA0hsyM75pDSxIU/qrS0sMHfZskrHxBGzeeNDmbXRjehaS/p8HCzWtJVVpyOgB hjwoi8ZDdkjgdv/g+JHn0sTn+xJM9L96m4K7zSvkwIobPJl0Gl95Xf98DQTlBVGu6NL9e2 P4T1heQ1Ya0CVUhakMTkEjDKBz8lsG0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657198391; a=rsa-sha256; cv=none; b=h5gb4F+nqI+t5M6GVfbyVEb24ewiUQ6Ay71uRe5IVzdGE2+1Stqc0pe4zhw5y7KnIIVU9u XGEXo0ZIa7vnItEe8kYEg20L+VP8vMdJdB5U6lBszOlpchpklc7IjnpheTHU53DdsUUQkE 6VRchQSggkFiZYjcS0Sc1ncTBRxli70= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=nuqQ+wJ5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-Stat-Signature: roddhbt5cb5k1yj8i41qxzuyrkixd3wy X-Rspamd-Queue-Id: CACEFC0057 X-Rspam-User: Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=nuqQ+wJ5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf22.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-Rspamd-Server: rspam10 X-HE-Tag: 1657198391-514110 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song This reverts commit 6bfef171d0d74cb050112e0e49feb20bfddf7f42. I was wrong to say batched tlb flush didn't apply to ARM. The fact is though ARM64 has hardware TLB flush, it is not free and could be quite expensive. We still have a good chance to enable batched and deferred TLB flush on ARM64 for memory reclamation. A possible way is that we only queue tlbi instructions in hardware's queue. When we have to broadcast TLB, we broadcast it by dsb. We just need to get adapted to the existing framework of BATCHED_UNMAP_TLB_FLUSH. Cc: Will Deacon Cc: Jonathan Corbet Cc: Nadav Amit Cc: Mel Gorman Signed-off-by: Barry Song --- Documentation/features/arch-support.txt | 1 - Documentation/features/vm/TLB/arch-support.txt | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/Documentation/features/arch-support.txt b/Documentation/features/arch-support.txt index 118ae031840b..d22a1095e661 100644 --- a/Documentation/features/arch-support.txt +++ b/Documentation/features/arch-support.txt @@ -8,5 +8,4 @@ The meaning of entries in the tables is: | ok | # feature supported by the architecture |TODO| # feature not yet supported by the architecture | .. | # feature cannot be supported by the hardware - | N/A| # feature doesn't apply to the architecture diff --git a/Documentation/features/vm/TLB/arch-support.txt b/Documentation/features/vm/TLB/arch-support.txt index 039e4e91ada3..1c009312b9c1 100644 --- a/Documentation/features/vm/TLB/arch-support.txt +++ b/Documentation/features/vm/TLB/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | N/A | + | arm64: | TODO | | csky: | TODO | | hexagon: | TODO | | ia64: | TODO | From patchwork Thu Jul 7 12:52:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 12909535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E798C433EF for ; Thu, 7 Jul 2022 12:53:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F7A96B0074; Thu, 7 Jul 2022 08:53:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A6886B0075; Thu, 7 Jul 2022 08:53:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86E7A6B0078; Thu, 7 Jul 2022 08:53:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 788126B0074 for ; Thu, 7 Jul 2022 08:53:19 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 4C12F80E1B for ; Thu, 7 Jul 2022 12:53:19 +0000 (UTC) X-FDA: 79660294518.05.F17D9DE Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf01.hostedemail.com (Postfix) with ESMTP id E0DEA40010 for ; Thu, 7 Jul 2022 12:53:18 +0000 (UTC) Received: by mail-pf1-f179.google.com with SMTP id a15so19147568pfv.13 for ; Thu, 07 Jul 2022 05:53:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KohxSBlyb2gPjZjQOuscJE9srXX9sKJcN9WDxnjJeNE=; b=Z/4G+j0uofz5mjT+m2wBBS33jHt4uCc2grw6vajzycA8dAivB2I1E1uYK99jJIfo3O +QVMHgjhpLcxM5rK9KyJLhGd/2RG7WG+m+aIuZ3X5Dg4EPoe2WAdSN9T1oNWhnIRZCdt ZtoFweBV2b0oTa0dw6ht0oMasl/SwPv6uthPPSyM3N0RJ0saHZLDz4W90bArUtgevKYX m4cfQNFxf1WMKIGk+78eEF2P//QSs/IM5W/9lyROXPgyN8uTHDD7SQNwWbsVEYeSgcNq xvvHmPTuceHpBwtGmhSs2cQ4RtAGjDNqkq+j6cmwe55l10OUsxpR+oO0AVJ7Iz/bxSVX mLOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KohxSBlyb2gPjZjQOuscJE9srXX9sKJcN9WDxnjJeNE=; b=sBTdpiW6X2cw93yHOWJ+8V/RWBP2NfGaTP5REaQaJ9q2FXQYv1UqUQDcNLD7dmuhtl Nhgm95JQfUJ7ruv2edxu8Aj4Sm+JfCbVXbFYGE2gjAMFjuBlbILxz7xAnUOK/PIu04E2 EOuV21VjmZEE+3DHFEC6Jlbx5sVYcQJ3sInWjwGRCImS5C173hTEy50y/+B8lFdDMy/y LdbGrSjHEdO6wKhk9MHxyUyQvNGmYtsajztmyfzntEfs+af3k8xZz8vCE1l9eGXN1FOE 70QWU8VCv0rZW+8cTa1gLwQZzkwyq5igf8t13wS2gkDJ9Z0KL6QInPn06PNu/SDOxdT+ 3H2A== X-Gm-Message-State: AJIora9wfszpy80gL4+KvWlCySHzsq2v0oPZx9yfUgMaNZExQzg3uD4A psX+PBpfztKqGmu/6JAu7MrG84K4jFRkLg== X-Google-Smtp-Source: AGRyM1tvGgyyQudlC4LStAUoiLkQLMqA8RlR3btbebQT65iLnPCQfEEP4u4looKoNOsGSkmoEtUiHg== X-Received: by 2002:a17:902:e809:b0:16a:22dc:d23a with SMTP id u9-20020a170902e80900b0016a22dcd23amr53568912plg.119.1657198397970; Thu, 07 Jul 2022 05:53:17 -0700 (PDT) Received: from localhost.localdomain (47-72-206-164.dsl.dyn.ihug.co.nz. [47.72.206.164]) by smtp.gmail.com with ESMTPSA id a11-20020a1709027e4b00b0016a3b5a46f0sm27907831pln.241.2022.07.07.05.53.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Jul 2022 05:53:17 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org Cc: corbet@lwn.net, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, yangyicong@hisilicon.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, Barry Song , Nadav Amit , Mel Gorman Subject: [PATCH 2/4] mm: rmap: Allow platforms without mm_cpumask to defer TLB flush Date: Fri, 8 Jul 2022 00:52:40 +1200 Message-Id: <20220707125242.425242-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220707125242.425242-1-21cnbao@gmail.com> References: <20220707125242.425242-1-21cnbao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657198399; a=rsa-sha256; cv=none; b=RZ2AKuL2BbaMvSVKwxcdmq9Cysua7cDGR4LxTEGsfnICsJxYL/eqstKn++kv0+pHcrh7zN /53LdXYcW4+P1K4Iyviwh0fw76JREk3tvUqY6w88njpzrAHuiI6Fpy/7yggAFJ2ymMiVj2 l+8rSr7x1YqzdA6L8gBwM1mRkxiNvbM= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="Z/4G+j0u"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657198399; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KohxSBlyb2gPjZjQOuscJE9srXX9sKJcN9WDxnjJeNE=; b=HrEWxrNw06fauB2Ruf21BSM5I5pDfqI+k6VwSngT68qHxeQezLMM82wUnTh0GBPhlbQCk+ jEF4GGW2AWemJmvxASg6xxzB9QNZehycKSwlc9VKGNnY5vTUYg4EfXTByP6+3UReQMunQr cxaFyAWOAE8XuUlqazFZrkbd7mV5F+U= X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E0DEA40010 X-Rspam-User: Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="Z/4G+j0u"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-Stat-Signature: dp8xoh68w87hre3663rtep3za3nstddg X-HE-Tag: 1657198398-583897 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song Platforms like ARM64 have hareware TLB shootdown broadcast. They don't maintain mm_cpumask and they just send tlbi and related sync instructions for TLB flush. So if mm_cpumask is empty, we also allow deferred TLB flush Cc: Nadav Amit Cc: Mel Gorman Signed-off-by: Barry Song > --- mm/rmap.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 5bcb334cd6f2..d320c29a4ad8 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -692,8 +692,13 @@ static bool should_defer_flush(struct mm_struct *mm, enum ttu_flags flags) if (!(flags & TTU_BATCH_FLUSH)) return false; - /* If remote CPUs need to be flushed then defer batch the flush */ - if (cpumask_any_but(mm_cpumask(mm), get_cpu()) < nr_cpu_ids) + /* + * If remote CPUs need to be flushed then defer batch the flush; + * If ARCHs like ARM64 have hardware TLB flush broadcast, thus + * they don't maintain mm_cpumask() at all, defer batch as well. + */ + if (cpumask_any_but(mm_cpumask(mm), get_cpu()) < nr_cpu_ids || + cpumask_empty(mm_cpumask(mm))) should_defer = true; put_cpu(); From patchwork Thu Jul 7 12:52:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 12909536 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89C70C43334 for ; Thu, 7 Jul 2022 12:53:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CB726B0075; Thu, 7 Jul 2022 08:53:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 27B096B0078; Thu, 7 Jul 2022 08:53:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 143D66B007B; Thu, 7 Jul 2022 08:53:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 06BE16B0075 for ; Thu, 7 Jul 2022 08:53:28 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id D4D1480A76 for ; Thu, 7 Jul 2022 12:53:27 +0000 (UTC) X-FDA: 79660294854.17.7BE09F7 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf01.hostedemail.com (Postfix) with ESMTP id 7ECE440010 for ; Thu, 7 Jul 2022 12:53:27 +0000 (UTC) Received: by mail-pg1-f171.google.com with SMTP id s27so18315573pga.13 for ; Thu, 07 Jul 2022 05:53:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6hOJXYJTQiFMMMc5LL8OXOjbAEOIGj7/DEojQqeqmtE=; b=nrNTLh5dPkoPJJ27EI8PkJlZZO7GsFZx7opX0T8UvM1J5eA0J0eT4vHf2MUS3ewfAl FYh/eMc6wEExO54NSw9YKO2kSHlBcinWZodF+x+U+oz7JI2X9cWHBmNGhFnlaRTQ70UD s765zJKqYUav6EwIEKQDKDpYCQ/4ZssXZLVwEesK4tDLlbrZTltsTvnGvPSgv55r0xuM bnGzeo+AVVRMW57URUjGLslcMHanIz5/piUmtJrCgf/QqZQwLbfdac8WuAmbUGFQt4qa 44J7UJkocdhXL3nGxV56UBmrUwnDVBBXswUTlKxuI/JvxncWSZZBW0tYexzHGBHQ5FnU F4Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6hOJXYJTQiFMMMc5LL8OXOjbAEOIGj7/DEojQqeqmtE=; b=3qi4CsoT0o5aCWHQ6HpQ4uWoyaI0Ak76wIROnLeWXNwtQoq3i/QQ6HRL48/ALw1F5e 2CGHkLpiSq300lRbVlGECxfg7c5NbSLnDNKI9uUrBc5X0S+UGM5pBXc5dcyiVDwBL/S5 CL7PVws+zJ2cyRvQX5anqPnZmoj6yrJmWflsardqYra9JCr0AzO/XQRsVYdaUmQc4h1P c9bs9vtL3pt+N3JxCoJf73WnyZxxsZ2edK4lYbyKGX7U1VfwiVbGGEs14iCbCpt2IaaM JmCh9xB0FK4aXVV8j+HSLTjDsh6/+OgzuuWy7RsGw2puhChXw72j0pei1d4Yeyb0iX3x CQhA== X-Gm-Message-State: AJIora8rxZNacQuU5gkO4N5MTQ0P8A3W+l8ynfmsR1XPvpCs+jUj53wD DFJ2ShIYSLgSnIFLmpNHZVA= X-Google-Smtp-Source: AGRyM1uZgm8qesTiPzIyn0Fdt/bFOX8tuNIMzYZlDgO6uvwzSy88a4q5p6lNgoDbQtEcMDOvySQXvg== X-Received: by 2002:a63:2115:0:b0:412:b292:1870 with SMTP id h21-20020a632115000000b00412b2921870mr4253648pgh.176.1657198406594; Thu, 07 Jul 2022 05:53:26 -0700 (PDT) Received: from localhost.localdomain (47-72-206-164.dsl.dyn.ihug.co.nz. [47.72.206.164]) by smtp.gmail.com with ESMTPSA id a11-20020a1709027e4b00b0016a3b5a46f0sm27907831pln.241.2022.07.07.05.53.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Jul 2022 05:53:26 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org Cc: corbet@lwn.net, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, yangyicong@hisilicon.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, Barry Song , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Nadav Amit , Mel Gorman Subject: [PATCH 3/4] mm: rmap: Extend tlbbatch APIs to fit new platforms Date: Fri, 8 Jul 2022 00:52:41 +1200 Message-Id: <20220707125242.425242-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220707125242.425242-1-21cnbao@gmail.com> References: <20220707125242.425242-1-21cnbao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657198407; a=rsa-sha256; cv=none; b=sNtfwxnona8Oa9BFZFShNfrFjSH8rF32OBaJkUGnRfC+lWe8nImQoyL4iLq4R3Z2mbF16q hlXQCv0vGUuG48sNfg7DfYPlVXQPbNd6JsL/urwYhHvQlyzp8KAiIx0fNrgLs6TG4IjGEm E+UysM+QsqV2pf68BCwNxJq4IVAuaxA= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=nrNTLh5d; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657198407; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6hOJXYJTQiFMMMc5LL8OXOjbAEOIGj7/DEojQqeqmtE=; b=P+T2FnminIiWCMDi2JjWIzzlEB0z5qv9l8iZeyjAh2XFbVCPXNOwgVjL+gwdeksHbdhh+w PnevTwMW6hWScd0k/LB+eq09QUhgwORcmaM8WVSREV0aVKe3u2j939QapaxTFlVW0O8Kxm Vns/+wQRh6UmoZ0JdSf5L+X+oJTg3zI= X-Stat-Signature: ch4roj6bmomhhfimeef3driw81ddj6iu X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7ECE440010 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=nrNTLh5d; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-HE-Tag: 1657198407-678718 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song Add vma and uaddr to tlbbatch APIs so that platforms like ARM64 are able to apply these two parameters on their specific hard- ware features. For ARM64, this could be sending tlbi into hard- ware queues without waiting for its completion. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Nadav Amit Cc: Mel Gorman Signed-off-by: Barry Song --- arch/x86/include/asm/tlbflush.h | 4 +++- mm/rmap.c | 12 ++++++++---- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 4af5579c7ef7..9fc48c103b31 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -251,7 +251,9 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) } static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm) + struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long uaddr) { inc_mm_tlb_gen(mm); cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); diff --git a/mm/rmap.c b/mm/rmap.c index d320c29a4ad8..2b5b740d0001 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -642,12 +642,14 @@ void try_to_unmap_flush_dirty(void) #define TLB_FLUSH_BATCH_PENDING_LARGE \ (TLB_FLUSH_BATCH_PENDING_MASK / 2) -static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable) +static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable, + struct vm_area_struct *vma, + unsigned long uaddr) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; int batch, nbatch; - arch_tlbbatch_add_mm(&tlb_ubc->arch, mm); + arch_tlbbatch_add_mm(&tlb_ubc->arch, mm, vma, uaddr); tlb_ubc->flush_required = true; /* @@ -737,7 +739,9 @@ void flush_tlb_batched_pending(struct mm_struct *mm) } } #else -static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable) +static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable, + struct vm_area_struct *vma, + unsigned long uaddr) { } @@ -1600,7 +1604,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pte_dirty(pteval)); + set_tlb_ubc_flush_pending(mm, pte_dirty(pteval), vma, address); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } From patchwork Thu Jul 7 12:52:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 12909537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42039C43334 for ; Thu, 7 Jul 2022 12:53:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA4E26B0074; Thu, 7 Jul 2022 08:53:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B54C96B0078; Thu, 7 Jul 2022 08:53:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1C426B007B; Thu, 7 Jul 2022 08:53:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 930756B0074 for ; Thu, 7 Jul 2022 08:53:35 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3BA22F83 for ; Thu, 7 Jul 2022 12:53:35 +0000 (UTC) X-FDA: 79660295190.08.FAF946A Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf05.hostedemail.com (Postfix) with ESMTP id C9C01100011 for ; Thu, 7 Jul 2022 12:53:34 +0000 (UTC) Received: by mail-pg1-f174.google.com with SMTP id r22so11442899pgr.2 for ; Thu, 07 Jul 2022 05:53:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l6cy31AaJrTdKl6fL2izFIU+5DjlDJpwVJm+i/nQovk=; b=Mw3+dLRsJC4e2NjNnGQ8GZOx403oBXgVn6D1eOVIPJvFVXSstetmEFSbhCsb8HhlUC RlYn7D6eaSp23TCtgAseziG3X2fBxSxj/39P0MOym+L5HtgW9NP205q+66DGL96v0D3T ax26+bD4qvd4oR1OL+ozcEEn1+ljrNCCrDgavzO5MPWV1M1qBltdfHBkH8PjOSffajyZ dVYRaWYd9aqqRewQYbslFTOUilBFaNa2ISPQGBDdGDTpOwl0lLRHqv8Ih/U0EMKeqQZ0 FmDeMiJs6n5MEiTa7rdcqpxrIRo3gb6+O9Bu6NkgNEBjZ5M5bnaQiQCQRDeNV160o+WQ reog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l6cy31AaJrTdKl6fL2izFIU+5DjlDJpwVJm+i/nQovk=; b=NJ/7tM3VGHI/OEGUsFSd66H0PABi2zB0RrwH+w1BHvEkGzm69SXZvLEgvOvge3wiQE mcK+qmbthW6RziyJfjBAUbP4YhT2wGFlmBo6adzkDmDBwOtiG3lKob9RwMzNxf34m/Mm PkijLksfQVAmlQyRWG7u12lWM3vBMmUjzqgRk9w6b6vRcBsQdZkGE71Fpv4H4iGZ2BHb RIk8IAhc0bp0hz6ORF39x9UeEZP1H4lC+U2f/TlHOXt7dIqkNWG+sLRITUDbRWAtUrS8 HfOa8bZRXzv8n9PbtYqcoSV54ztsWGe2ULFySBPFQob/hIV/i4VgIUh/zvRekRD098Ya t5fA== X-Gm-Message-State: AJIora9VAJEu8DOAb0KT4PeLfa4BLT4cIxZEXcEmTVemPBlworVm0qdF /PonEwuGUG3wARwB1WvoZVA= X-Google-Smtp-Source: AGRyM1sF8g/5GPUGhwsID/TrPiRGfdyrAfdsWsMgLol9kbQ1rpat6WTfw2GLXYGMztMfhQ8sIIIhaw== X-Received: by 2002:a63:6d4c:0:b0:414:ec27:2c87 with SMTP id i73-20020a636d4c000000b00414ec272c87mr1348336pgc.495.1657198413868; Thu, 07 Jul 2022 05:53:33 -0700 (PDT) Received: from localhost.localdomain (47-72-206-164.dsl.dyn.ihug.co.nz. [47.72.206.164]) by smtp.gmail.com with ESMTPSA id a11-20020a1709027e4b00b0016a3b5a46f0sm27907831pln.241.2022.07.07.05.53.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Jul 2022 05:53:33 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org Cc: corbet@lwn.net, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, yangyicong@hisilicon.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, Barry Song , Nadav Amit , Mel Gorman Subject: [PATCH 4/4] arm64: support batched/deferred tlb shootdown during page reclamation Date: Fri, 8 Jul 2022 00:52:42 +1200 Message-Id: <20220707125242.425242-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220707125242.425242-1-21cnbao@gmail.com> References: <20220707125242.425242-1-21cnbao@gmail.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657198414; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l6cy31AaJrTdKl6fL2izFIU+5DjlDJpwVJm+i/nQovk=; b=7SfVi6JiWZc16MZdMAGE3PiOc2wbV418KtP/U17ROsnj8xk0qyrE9sAicPhepLWeZzTNR3 V4SoqxJpdSFf8K7EAJ53Cx162lmTzSnCcVvrvtq2Q4Jc0fudTifOEMRImzVSdPwijnKc5O /LTHCom3J04n3R6Z8uF7Z1K96WzEb3U= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657198414; a=rsa-sha256; cv=none; b=8EstF35jGIlCWwXXvuuhiF5reLLG1J+1n8PPho/jo6KxPARrH5MZDTq8GRUsC1m0fTYOav An6ZTzir1SAEXOqkWkNExH8wVqO43A60vYd01QDdxgROUtRmFEKATR4R7ABe2jRAXWC0OU +bn4YDnAymqkwSlBiAJxb3xbSyb3CeI= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Mw3+dLRs; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-Stat-Signature: htjpqbt5xa7otjsaey1wpk1qp748wafa X-Rspamd-Queue-Id: C9C01100011 X-Rspam-User: Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Mw3+dLRs; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-Rspamd-Server: rspam10 X-HE-Tag: 1657198414-67496 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song on x86, batched and deferred tlb shootdown has lead to 90% performance increase on tlb shootdown. on arm64, HW can do tlb shootdown without software IPI. But sync tlbi is still quite expensive. Even running a simplest program which requires swapout can prove this is true, #include #include #include #include int main() { #define SIZE (1 * 1024 * 1024) volatile unsigned char *p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); memset(p, 0x88, SIZE); for (int k = 0; k < 10000; k++) { /* swap in */ for (int i = 0; i < SIZE; i += 4096) { (void)p[i]; } /* swap out */ madvise(p, SIZE, MADV_PAGEOUT); } } Perf result on snapdragon 888 with 8 cores by using zRAM as the swap block device. ~ # perf record taskset -c 4 ./a.out [ perf record: Woken up 10 times to write data ] [ perf record: Captured and wrote 2.297 MB perf.data (60084 samples) ] ~ # perf report # To display the perf.data header info, please use --header/--header-only options. # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 60K of event 'cycles' # Event count (approx.): 35706225414 # # Overhead Command Shared Object Symbol # ........ ....... ................. ............................................................................. # 21.07% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irq 8.23% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore 6.67% a.out [kernel.kallsyms] [k] filemap_map_pages 6.16% a.out [kernel.kallsyms] [k] __zram_bvec_write 5.36% a.out [kernel.kallsyms] [k] ptep_clear_flush 3.71% a.out [kernel.kallsyms] [k] _raw_spin_lock 3.49% a.out [kernel.kallsyms] [k] memset64 1.63% a.out [kernel.kallsyms] [k] clear_page 1.42% a.out [kernel.kallsyms] [k] _raw_spin_unlock 1.26% a.out [kernel.kallsyms] [k] mod_zone_state.llvm.8525150236079521930 1.23% a.out [kernel.kallsyms] [k] xas_load 1.15% a.out [kernel.kallsyms] [k] zram_slot_lock ptep_clear_flush() takes 5.36% CPU in the micro-benchmark swapping in/out a page mapped by only one process. If the page is mapped by multiple processes, typically, like more than 100 on a phone, the overhead would be much higher as we have to run tlb flush 100 times for one single page. Plus, tlb flush overhead will increase with the number of CPU cores due to the bad scalability of tlb shootdown in HW, so those ARM64 servers should expect much higher overhead. Further perf annonate shows 95% cpu time of ptep_clear_flush is actually used by the final dsb() to wait for the completion of tlb flush. This provides us a very good chance to leverage the existing batched tlb in kernel. The minimum modification is that we only send async tlbi in the first stage and we send dsb while we have to sync in the second stage. With the above simplest micro benchmark, collapsed time to finish the program decreases around 5%. Typical collapsed time w/o patch: ~ # time taskset -c 4 ./a.out 0.21user 14.34system 0:14.69elapsed w/ patch: ~ # time taskset -c 4 ./a.out 0.22user 13.45system 0:13.80elapsed Cc: Jonathan Corbet Cc: Nadav Amit Cc: Mel Gorman Signed-off-by: Barry Song Tested-by: Yicong Yang --- Documentation/features/vm/TLB/arch-support.txt | 2 +- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/tlbbatch.h | 12 ++++++++++++ arch/arm64/include/asm/tlbflush.h | 13 +++++++++++++ 4 files changed, 27 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/include/asm/tlbbatch.h diff --git a/Documentation/features/vm/TLB/arch-support.txt b/Documentation/features/vm/TLB/arch-support.txt index 1c009312b9c1..2caf815d7c6c 100644 --- a/Documentation/features/vm/TLB/arch-support.txt +++ b/Documentation/features/vm/TLB/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | TODO | + | arm64: | ok | | csky: | TODO | | hexagon: | TODO | | ia64: | TODO | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1652a9800ebe..e94913a0b040 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -93,6 +93,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/include/asm/tlbbatch.h b/arch/arm64/include/asm/tlbbatch.h new file mode 100644 index 000000000000..fedb0b87b8db --- /dev/null +++ b/arch/arm64/include/asm/tlbbatch.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ARCH_ARM64_TLBBATCH_H +#define _ARCH_ARM64_TLBBATCH_H + +struct arch_tlbflush_unmap_batch { + /* + * For arm64, HW can do tlb shootdown, so we don't + * need to record cpumask for sending IPI + */ +}; + +#endif /* _ARCH_ARM64_TLBBATCH_H */ diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25..b3ed163267ca 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -272,6 +272,19 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, dsb(ish); } +static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long uaddr) +{ + flush_tlb_page_nosync(vma, uaddr); +} + +static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) +{ + dsb(ish); +} + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement.