From patchwork Mon Jul 11 03:46:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 12912782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCF7EC43334 for ; Mon, 11 Jul 2022 03:46:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A2548E0003; Sun, 10 Jul 2022 23:46:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 42A8A8E0001; Sun, 10 Jul 2022 23:46:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CC108E0003; Sun, 10 Jul 2022 23:46:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1E66D8E0001 for ; Sun, 10 Jul 2022 23:46:48 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CA18720560 for ; Mon, 11 Jul 2022 03:46:47 +0000 (UTC) X-FDA: 79673432454.24.01846AE Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf01.hostedemail.com (Postfix) with ESMTP id 6EEAC40029 for ; Mon, 11 Jul 2022 03:46:47 +0000 (UTC) Received: by mail-pj1-f51.google.com with SMTP id t5-20020a17090a6a0500b001ef965b262eso3772754pjj.5 for ; Sun, 10 Jul 2022 20:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=zH1rOKwt+eppc4uRJ3wzEb3jlVxvSJQcRh3p/hoLlMo=; b=ePx9tw2qDorWxYzPWcGRLOvGkwfDV4WFpjno1bzwsIkpBxrJHAnbnXDd+9TSQPIZWc rB8vctJmdtvbZmL7h6HXuBkhUVLcRj4hr5D8Xao8ADv5y3VOodNZydMQYF/4GMGusm5B AITvvCsvM4Fymc4mIY+V5NWWdLdN5dTrkn7xmybz4zKiKBZrzDwmWdMdWUAOQVASHsYU cWoYrJVQQwEh8W4X1CL7RJ+zQhdG9+qfpNmkK52I5z3wiRZLGNEEimJp5XIALeYrky8b md0dtPDDjYyabBKb/WuxSnQ+6lF4nSZ8gZYRptwvwsM03yDen/D8SIE11lqmoUisCWjs XneA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zH1rOKwt+eppc4uRJ3wzEb3jlVxvSJQcRh3p/hoLlMo=; b=WZjxMA066Fnopk5QpVJY22PbuvyatABS4CbCfOVmeyEwVOoTU/Ox8iZ3Gi+oB9QDe0 cbb22U78OEgDgPe9X9KT7KKp/i5Sam20X/ftK0hdZ6lV2SWWz4FAJ+PA6vSuADCgKhEA p3bn2UAyejz1JMlQF4VtoFRcjXQDTGi/ego/Lzsk2LsAmznmFFJBT6l97NMpYDQo+q5r NSf/mOoX0IOoTYFPpv9sy79y/U4L+uCa5lpb5Qdzv2S54t0fRxgiBwC4sj9EnwaBghnR +0LEV3TJ41TpDQnClIEznEgqlyZkoGgSYAZjhjuIILDlh9Y1wPghk9wrqfRaigAEm6cE PWHA== X-Gm-Message-State: AJIora81xITxhGnnUAaHzINFKjeMzkA+nvRRrgHMTaGdbLmvdG9J7rpR kq4CjKLZC7ctENV84u7HOHE= X-Google-Smtp-Source: AGRyM1t+IjLE84wZXwipeZB/XUo7NKGO7BarDvcoewmact2teddnhHzQthXl8IiBONeN1GAjHbnetA== X-Received: by 2002:a17:90b:4b4a:b0:1ef:fc95:3c4f with SMTP id mi10-20020a17090b4b4a00b001effc953c4fmr15058701pjb.138.1657511206386; Sun, 10 Jul 2022 20:46:46 -0700 (PDT) Received: from localhost.localdomain (47-72-206-164.dsl.dyn.ihug.co.nz. [47.72.206.164]) by smtp.gmail.com with ESMTPSA id a13-20020a170902eccd00b001664d88aab3sm3447949plh.240.2022.07.10.20.46.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:46:45 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org Cc: corbet@lwn.net, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, yangyicong@hisilicon.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Barry Song Subject: [PATCH v2 1/4] Revert "Documentation/features: mark BATCHED_UNMAP_TLB_FLUSH doesn't apply to ARM64" Date: Mon, 11 Jul 2022 15:46:12 +1200 Message-Id: <20220711034615.482895-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220711034615.482895-1-21cnbao@gmail.com> References: <20220711034615.482895-1-21cnbao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657511207; a=rsa-sha256; cv=none; b=seSzfYnTVJabjnAxdP4DWf34UnJUw3NzaQBU6XxX4V5JZvWmt4p++pDxue3hlaZpDkgcw6 L/OZY1nXUy7Fotvoa8FSfE4hAPjtrYDI2l3BpACsn4x8TSHfYL9a//pMB7V1Li/9DQVfkU Dzowj27EklLWnbEKBECFsrUt1c/QDqM= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ePx9tw2q; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657511207; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zH1rOKwt+eppc4uRJ3wzEb3jlVxvSJQcRh3p/hoLlMo=; b=OaaLDpKJKtpPtpB+YQcnmorEIJt0Z3rL8RKZUMBf7XnKFSfDdGgj2q6pHZ3Egu4ecTtTIz R7AEsuBwFAbPr4ZANysKg0iBWxWmL9d3aMwUjIuib5V3sa4JavDerSZd0wm4FsfVpCcGUn nfsbZbmzzswLY4u8DhaqyLTKhDngQVQ= X-Stat-Signature: tqhsk4fa6b43bimqiw3ky79oujsq4iei Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=ePx9tw2q; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6EEAC40029 X-Rspam-User: X-HE-Tag: 1657511207-586259 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song This reverts commit 6bfef171d0d74cb050112e0e49feb20bfddf7f42. I was wrong. Though ARM64 has hardware TLB flush, but it is not free and it is still expensive. We still have a good chance to enable batched and deferred TLB flush on ARM64 for memory reclamation. A possible way is that we only queue tlbi instructions in hardware's queue. When we have to broadcast TLB, we broadcast it by dsb. We just need to get adapted the existing BATCHED_UNMAP_TLB_FLUSH. Signed-off-by: Barry Song --- Documentation/features/arch-support.txt | 1 - Documentation/features/vm/TLB/arch-support.txt | 2 +- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/Documentation/features/arch-support.txt b/Documentation/features/arch-support.txt index 118ae031840b..d22a1095e661 100644 --- a/Documentation/features/arch-support.txt +++ b/Documentation/features/arch-support.txt @@ -8,5 +8,4 @@ The meaning of entries in the tables is: | ok | # feature supported by the architecture |TODO| # feature not yet supported by the architecture | .. | # feature cannot be supported by the hardware - | N/A| # feature doesn't apply to the architecture diff --git a/Documentation/features/vm/TLB/arch-support.txt b/Documentation/features/vm/TLB/arch-support.txt index 039e4e91ada3..1c009312b9c1 100644 --- a/Documentation/features/vm/TLB/arch-support.txt +++ b/Documentation/features/vm/TLB/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | N/A | + | arm64: | TODO | | csky: | TODO | | hexagon: | TODO | | ia64: | TODO | From patchwork Mon Jul 11 03:46:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 12912783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83A20C433EF for ; Mon, 11 Jul 2022 03:46:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D17E8E0005; Sun, 10 Jul 2022 23:46:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 05B118E0001; Sun, 10 Jul 2022 23:46:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E19308E0005; Sun, 10 Jul 2022 23:46:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D05D38E0001 for ; Sun, 10 Jul 2022 23:46:55 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id B0780120234 for ; Mon, 11 Jul 2022 03:46:55 +0000 (UTC) X-FDA: 79673432790.17.81094F4 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf01.hostedemail.com (Postfix) with ESMTP id 50A8E40026 for ; Mon, 11 Jul 2022 03:46:55 +0000 (UTC) Received: by mail-pf1-f174.google.com with SMTP id x184so3761111pfx.2 for ; Sun, 10 Jul 2022 20:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lXaZY9eIuwP2fJopKwfWWT3qEb555n6mJNYuev1dBus=; b=SioxnzTe2ch2xEYudKZfsl67d1QZ4TMyJtotZWTK0aAwzTLr9jj8xZp+pV2EbQoBT0 MAqRJsKlAj448br5lowpUBxxNEKiSIaSf4cPCV8M8MxZHjLrxh3lyCi8iKO61foW78+D U3wLwVIJShgAZq8T6IAOO45Y0W77tixdThRHmTwLzJ6R7JfZwOteG+R57W/788gzDSuM WSrrUIfKaYTC2JJcXRmfHiExmrpWtn7on9YUz3BRc1kulVxqkGhEPRDg5Zd7mbkhETJx kb2MG90IEqRvvepP4MKGh9DxFmcY/93lXsU9TywbTvUJeRXaVm5GeyLXgoa+sw77yvXJ swhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lXaZY9eIuwP2fJopKwfWWT3qEb555n6mJNYuev1dBus=; b=JvjTKWPzBZvpG7C1e6IzR8QS4ws5B1iOI33bPcwLJSyQ/gfcrWSnhmh14f7+tX98eS 98xHMhdst/doxx3L696GwVFsuUmZWR1r07DcH7DOW/G3Y373+IZARZKT+9oIf2xNTApI m6BP9CLQTh9dMUitP3hVqhNxjOxce4aiyBGmwEXTcxzcE2WnFiNhh7q3BxSaK7NvXRJL IxzaRtmIgiUBtQsiOUmOR6SBj/fx7+Bs35mcv2SCsgFnfbgcGhN/RLgpG9NjrPn7DYq4 TJRoimtZ2xtQhbeA1alrasuX46rwATHxDfa3dXQLPkcE5buVsyi6HAev5jBXfAYm45wY c3FA== X-Gm-Message-State: AJIora+ta83DhaHKrTOIUcG0P9YbC4iMRpNqfUQZY9AhfEl8Qiv0IKqm RF7Z6w4Ue14rv9432D9GR8dClGo41jdr8A== X-Google-Smtp-Source: AGRyM1sZQM/y1E9SIJsc5yJdrFZSjEkOM1rd5dQPXVIU+O/k1albRoiLAbJq418n0LjL4QbX4nTFhg== X-Received: by 2002:a05:6a00:1496:b0:52a:c3fb:8ec7 with SMTP id v22-20020a056a00149600b0052ac3fb8ec7mr6880047pfu.25.1657511214461; Sun, 10 Jul 2022 20:46:54 -0700 (PDT) Received: from localhost.localdomain (47-72-206-164.dsl.dyn.ihug.co.nz. [47.72.206.164]) by smtp.gmail.com with ESMTPSA id a13-20020a170902eccd00b001664d88aab3sm3447949plh.240.2022.07.10.20.46.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:46:53 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org Cc: corbet@lwn.net, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, yangyicong@hisilicon.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Barry Song Subject: [PATCH v2 2/4] mm: rmap: Allow platforms without mm_cpumask to defer TLB flush Date: Mon, 11 Jul 2022 15:46:13 +1200 Message-Id: <20220711034615.482895-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220711034615.482895-1-21cnbao@gmail.com> References: <20220711034615.482895-1-21cnbao@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657511215; a=rsa-sha256; cv=none; b=0kCMD/O41mNvw1PJwd9+PxKesiJUr8gfR3FaGWzUCvxCJCfm03RnUVTOxH0raVgyhXj388 KdW4OgVDDvd7k4KUzOaOYIuOsF8KG32RJ+P51D4NHYLVMjO8mp1a0Q2gAR+8XcRk86BPz+ 85DrSczgNmQ3m52tdpJll8KwjPglEBo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=SioxnzTe; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657511215; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lXaZY9eIuwP2fJopKwfWWT3qEb555n6mJNYuev1dBus=; b=3YhtfY97yPaUkypDjfHl5pQXLVG2BSltnPteDYdTfpqdCx1VWmKx4WgUA9AxYOSCY8pP0a tWI0fGvCWEm8dnwnGBw1AfqZFRi3CNoXZiaMTJ6Xv3rm4nJQzWISfwPwJ1CJZ6WIq3F6CQ TYYyfUsf+3Jz50K8DAWfTTr6JYr8JnE= X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 50A8E40026 X-Rspam-User: X-Stat-Signature: px3dea81kjmdc6djzi331yqeybjymikp Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=SioxnzTe; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-HE-Tag: 1657511215-512853 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song Platforms like ARM64 have hareware TLB shootdown broadcast. They don't maintain mm_cpumask but just send tlbi and related sync instructions for TLB flush. task's mm_cpumask is normally empty in this case. We also allow deferred TLB flush on this kind of platforms. Signed-off-by: Barry Song > --- arch/arm/Kconfig | 1 + arch/loongarch/Kconfig | 1 + arch/mips/Kconfig | 1 + arch/openrisc/Kconfig | 1 + arch/powerpc/Kconfig | 1 + arch/riscv/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/um/Kconfig | 1 + arch/x86/Kconfig | 1 + mm/Kconfig | 3 +++ mm/rmap.c | 4 ++++ 11 files changed, 16 insertions(+) diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 7630ba9cb6cc..25c42747f488 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -13,6 +13,7 @@ config ARM select ARCH_HAS_KEEPINITRD select ARCH_HAS_KCOV select ARCH_HAS_MEMBARRIER_SYNC_CORE + select ARCH_HAS_MM_CPUMASK select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PTE_SPECIAL if ARM_LPAE select ARCH_HAS_PHYS_TO_DMA diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig index 1920d52653b4..4b737c0d17a2 100644 --- a/arch/loongarch/Kconfig +++ b/arch/loongarch/Kconfig @@ -7,6 +7,7 @@ config LOONGARCH select ARCH_ENABLE_MEMORY_HOTPLUG select ARCH_ENABLE_MEMORY_HOTREMOVE select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI + select ARCH_HAS_MM_CPUMASK select ARCH_HAS_PHYS_TO_DMA select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index db09d45d59ec..1b196acdeca3 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -9,6 +9,7 @@ config MIPS select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_KCOV select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE if !EVA + select ARCH_HAS_MM_CPUMASK select ARCH_HAS_PTE_SPECIAL if !(32BIT && CPU_HAS_RIXI) select ARCH_HAS_STRNCPY_FROM_USER select ARCH_HAS_STRNLEN_USER diff --git a/arch/openrisc/Kconfig b/arch/openrisc/Kconfig index e814df4c483c..82483b192f4a 100644 --- a/arch/openrisc/Kconfig +++ b/arch/openrisc/Kconfig @@ -9,6 +9,7 @@ config OPENRISC select ARCH_32BIT_OFF_T select ARCH_HAS_DMA_SET_UNCACHED select ARCH_HAS_DMA_CLEAR_UNCACHED + select ARCH_HAS_MM_CPUMASK select ARCH_HAS_SYNC_DMA_FOR_DEVICE select COMMON_CLK select OF diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index c2ce2e60c8f0..19061ffe73a0 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -127,6 +127,7 @@ config PPC select ARCH_HAS_MEMBARRIER_SYNC_CORE select ARCH_HAS_MEMREMAP_COMPAT_ALIGN if PPC_64S_HASH_MMU select ARCH_HAS_MMIOWB if PPC64 + select ARCH_HAS_MM_CPUMASK select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PHYS_TO_DMA select ARCH_HAS_PMEM_API diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index c22f58155948..7570c95a9cc8 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -25,6 +25,7 @@ config RISCV select ARCH_HAS_GIGANTIC_PAGE select ARCH_HAS_KCOV select ARCH_HAS_MMIOWB + select ARCH_HAS_MM_CPUMASK select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SET_DIRECT_MAP if MMU select ARCH_HAS_SET_MEMORY if MMU diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index 91c0b80a8bf0..48d91fa05bab 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -73,6 +73,7 @@ config S390 select ARCH_HAS_GIGANTIC_PAGE select ARCH_HAS_KCOV select ARCH_HAS_MEM_ENCRYPT + select ARCH_HAS_MM_CPUMASK select ARCH_HAS_PTE_SPECIAL select ARCH_HAS_SCALED_CPUTIME select ARCH_HAS_SET_MEMORY diff --git a/arch/um/Kconfig b/arch/um/Kconfig index 4ec22e156a2e..df29c729267b 100644 --- a/arch/um/Kconfig +++ b/arch/um/Kconfig @@ -8,6 +8,7 @@ config UML select ARCH_EPHEMERAL_INODES select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_KCOV + select ARCH_HAS_MM_CPUMASK select ARCH_HAS_STRNCPY_FROM_USER select ARCH_HAS_STRNLEN_USER select ARCH_NO_PREEMPT diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index be0b95e51df6..a91d73866238 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -81,6 +81,7 @@ config X86 select ARCH_HAS_KCOV if X86_64 select ARCH_HAS_MEM_ENCRYPT select ARCH_HAS_MEMBARRIER_SYNC_CORE + select ARCH_HAS_MM_CPUMASK select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 diff --git a/mm/Kconfig b/mm/Kconfig index 169e64192e48..7bf54f57ca01 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -951,6 +951,9 @@ config ARCH_HAS_CURRENT_STACK_POINTER register alias named "current_stack_pointer", this config can be selected. +config ARCH_HAS_MM_CPUMASK + bool + config ARCH_HAS_VM_GET_PAGE_PROT bool diff --git a/mm/rmap.c b/mm/rmap.c index 5bcb334cd6f2..13d4f9a1d4f1 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -692,6 +692,10 @@ static bool should_defer_flush(struct mm_struct *mm, enum ttu_flags flags) if (!(flags & TTU_BATCH_FLUSH)) return false; +#ifndef CONFIG_ARCH_HAS_MM_CPUMASK + return true; +#endif + /* If remote CPUs need to be flushed then defer batch the flush */ if (cpumask_any_but(mm_cpumask(mm), get_cpu()) < nr_cpu_ids) should_defer = true; From patchwork Mon Jul 11 03:46:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 12912784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86E4CC43334 for ; Mon, 11 Jul 2022 03:47:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 258D58E0006; Sun, 10 Jul 2022 23:47:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 207C68E0001; Sun, 10 Jul 2022 23:47:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A9148E0006; Sun, 10 Jul 2022 23:47:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EDC148E0001 for ; Sun, 10 Jul 2022 23:47:05 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C267C20560 for ; Mon, 11 Jul 2022 03:47:05 +0000 (UTC) X-FDA: 79673433210.28.AC499CB Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf07.hostedemail.com (Postfix) with ESMTP id 6DF9C40035 for ; Mon, 11 Jul 2022 03:47:05 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id y14-20020a17090a644e00b001ef775f7118so7099294pjm.2 for ; Sun, 10 Jul 2022 20:47:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5IhrKX64Q/Z6V/1v+K+JVKP0KaAJzbrmRGjpirKbhJY=; b=nTf9L3OC40YJpmHflFSmz6KTAFSTaaZQ/S1ioxVsgmxptGp0IiaRp8EmBLD2A/gyAa xCxhqU4FrY/lAvJTq2moA9lgpqjoUbWVxt9jLxPCj3p706ca9EfduK+9ZEqASUp+iSn4 EAQGxSVUN0yLRZXgqjdbfj5Fmtrix3Zcu+LFvg4y16WF7hkCA9xAsOqzQxEjWd454u2h XE+9+f9Iud7s2Mj28twbgXcE2SL3KUqGtSOKhI9wUKt1VHNuIkvy6yU2jFaWI5VZAN/G f94VxtVlrnt015VrVBiB2PQUM0RSK74qAsmnUwlhXwOzhhh3WvHkY5vJB3bG/9C0lVRi fgUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5IhrKX64Q/Z6V/1v+K+JVKP0KaAJzbrmRGjpirKbhJY=; b=genenXyZ5JVAWJEk9OAxry5AiURZd4OFY6LGosetPNJgSNFXNHIfE8pu4lu/X5HWS9 uVra29rsUZvr/A5U6pBy3mosu9aXnprglTqo5dtKXr9Pihju41QOjgxGkFF+19nNGxg9 MMt01L/JOM4Vuw1MmCtad2E53XTFUcO8qL/mAMw76grR1cIOMAg5sjhEf9diaZ2BcxXY 3STUXt4Ksv+HO7j/zSZAHVMRR+Pc1TEn0gOWE6bbkeDO3uqm43oGjJQFBM/Wg1VRXkul suvV/9eJeUsq49/m8wsCkCfXXPKSb/Gg52Q8LpdMNamSHCbwLLjcjqq6FWR986geRx8a 3Xjg== X-Gm-Message-State: AJIora/5FHKzoD9fSrV3Rp/dwbP7bQHvNVh66hGWeVs/gDUIo8i6ZCe5 7AMkhX70UhCi70pbzzlYyWE= X-Google-Smtp-Source: AGRyM1tGbLroqairLqOJOtLQAbkOgD2oJX4eg7rNXG45d2PB1rwtsYjv7Fwlc1lyPAUr+54PmfGNUg== X-Received: by 2002:a17:902:8344:b0:16a:6e99:de2c with SMTP id z4-20020a170902834400b0016a6e99de2cmr16521282pln.130.1657511224541; Sun, 10 Jul 2022 20:47:04 -0700 (PDT) Received: from localhost.localdomain (47-72-206-164.dsl.dyn.ihug.co.nz. [47.72.206.164]) by smtp.gmail.com with ESMTPSA id a13-20020a170902eccd00b001664d88aab3sm3447949plh.240.2022.07.10.20.46.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:47:04 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org Cc: corbet@lwn.net, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, yangyicong@hisilicon.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Barry Song , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Nadav Amit , Mel Gorman Subject: [PATCH v2 3/4] mm: rmap: Extend tlbbatch APIs to fit new platforms Date: Mon, 11 Jul 2022 15:46:14 +1200 Message-Id: <20220711034615.482895-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220711034615.482895-1-21cnbao@gmail.com> References: <20220711034615.482895-1-21cnbao@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=nTf9L3OC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657511225; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5IhrKX64Q/Z6V/1v+K+JVKP0KaAJzbrmRGjpirKbhJY=; b=LPuUUlHqss/F5/RrK3xQDv8qDTFg5V+FYmShgv6KXbTq6PT3E4MsU1yefCXHyt+Oo7y7rq RW2bMr9cBRg7uH41dtL+fNvofUuaRYm8aNeTegvyXICqdrJzhV/A+mvxNVelX6gpuUbFRd NoqbbMeOGE9fLFFNPcECE3Jm53Lhubs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657511225; a=rsa-sha256; cv=none; b=lscprC/4kD0hmhdHNN/v22yb32Kai2uHLcoQxQr2y63LohYyBaWw9z/dNMPogeNQ1sRclV xWAWOYhF6yxftBu4d+Dd6611WgglGE/3uJ/1rsUz5H2jw/kL//Zh/pazdi+GEQI6Vo4gro 7C1vTNsnzzJ4XiEtV2pOsOpgObmq5hc= X-Rspam-User: Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=nTf9L3OC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.43 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-Stat-Signature: bdn8cxb7q7oqmh5aao7qgyjaum5umcrx X-Rspamd-Queue-Id: 6DF9C40035 X-Rspamd-Server: rspam04 X-HE-Tag: 1657511225-267968 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song Add uaddr to tlbbatch APIs so that platforms like ARM64 are able to apply this on their specific hardware features. For ARM64, this could be sending tlbi into hardware queues for the page with this particular uaddr. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: "H. Peter Anvin" Cc: Nadav Amit Cc: Mel Gorman Signed-off-by: Barry Song --- arch/x86/include/asm/tlbflush.h | 3 ++- mm/rmap.c | 10 ++++++---- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 4af5579c7ef7..1b32f4b999c7 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -251,7 +251,8 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) } static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm) + struct mm_struct *mm, + unsigned long uaddr) { inc_mm_tlb_gen(mm); cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); diff --git a/mm/rmap.c b/mm/rmap.c index 13d4f9a1d4f1..a52381a680db 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -642,12 +642,13 @@ void try_to_unmap_flush_dirty(void) #define TLB_FLUSH_BATCH_PENDING_LARGE \ (TLB_FLUSH_BATCH_PENDING_MASK / 2) -static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable) +static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable, + unsigned long uaddr) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; int batch, nbatch; - arch_tlbbatch_add_mm(&tlb_ubc->arch, mm); + arch_tlbbatch_add_mm(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; /* @@ -736,7 +737,8 @@ void flush_tlb_batched_pending(struct mm_struct *mm) } } #else -static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable) +static void set_tlb_ubc_flush_pending(struct mm_struct *mm, bool writable, + unsigned long uaddr) { } @@ -1599,7 +1601,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pte_dirty(pteval)); + set_tlb_ubc_flush_pending(mm, pte_dirty(pteval), address); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } From patchwork Mon Jul 11 03:46:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 12912785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77D5EC433EF for ; Mon, 11 Jul 2022 03:47:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8DEB8E0007; Sun, 10 Jul 2022 23:47:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E3CC38E0001; Sun, 10 Jul 2022 23:47:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CDF678E0007; Sun, 10 Jul 2022 23:47:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BEE728E0001 for ; Sun, 10 Jul 2022 23:47:14 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8F3F034358 for ; Mon, 11 Jul 2022 03:47:14 +0000 (UTC) X-FDA: 79673433588.21.733C2A6 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf18.hostedemail.com (Postfix) with ESMTP id 248A51C0069 for ; Mon, 11 Jul 2022 03:47:13 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id 5so3392706plk.9 for ; Sun, 10 Jul 2022 20:47:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kX77QQPavnlmqTPJr+cPH2hNKZO7UlqRRz965T9dDrs=; b=htIXMTHV1qfaPfX3l4XZ4hj+VaUJoyeJxf3BRtOKXsqpYicKtvT6e4VttVAI3DicXN 9gM6V/tgXsGAqEJx6fyT+PeWPFlSYsi+4M5SHPDGMW/jL34jf6NoLM0kYfBinO8DT1MR DPdLnhEyAt7VslHXHkcl8xE/RUV8tJufwFlQVmtY3LpgfPw7A4qLCzw/q/dCdkea/U/H el05aKwmbkbJfRsPZasOu7jfAx2d4VJiPcyDUNvV41zJXsh6JXPpax8tmfD7C5+a9ygQ wOaqM8gP36qTLyaFt1jXVEk29x4zttAJoOdz3yRBuZA7IPt8psRkaoiAygE7YyPtwQ1Z E2zQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kX77QQPavnlmqTPJr+cPH2hNKZO7UlqRRz965T9dDrs=; b=5bngT6qTVXed3QF8niCPTwFr2QAMHiD35HUrcSR8HW0Z9PtOsEGr+RHMDvKrcwkyKC naYg5WmRBBolAuUVAYKObMtcpJ18YPaqWiWJYHjr6YW9uX3VlpxJ8nx94B7I3npyGs2s VsXobyxeAuzsjESD79RE/p5NfY3s15U4U044Ne+MzSKR+45xa6LbNhDx00KF/dbP+Lqn +oRBrKHar/E/hKElyVX3aHMXYa8Qr+WXRT9gYVrpwDPMfZ16Ld/HGp+MxzHyXQaHV82/ NzUeZSJqjHwFUULA/kygmP++VzjnJx0Z5IkMJGyXpeWQq0Hf6nZ0Kke2UYODHRgFvfOD mULw== X-Gm-Message-State: AJIora8I/6t21ngKPzWFIB4L3e1xdQRv3JNun4p/Is6jWUYsnxYczo6k F+RNm8vjwpi5U6P9Oo9KzZE= X-Google-Smtp-Source: AGRyM1vLe/YZvbOkUYTH5iAqm8VRsVnu2Afj8VTLOnXJSTfR2PgdfzJzSBQsk0Iq2lETGsM8bxMYpQ== X-Received: by 2002:a17:90b:1c90:b0:1ee:d804:d2c1 with SMTP id oo16-20020a17090b1c9000b001eed804d2c1mr15473364pjb.92.1657511233142; Sun, 10 Jul 2022 20:47:13 -0700 (PDT) Received: from localhost.localdomain (47-72-206-164.dsl.dyn.ihug.co.nz. [47.72.206.164]) by smtp.gmail.com with ESMTPSA id a13-20020a170902eccd00b001664d88aab3sm3447949plh.240.2022.07.10.20.47.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:47:12 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org Cc: corbet@lwn.net, arnd@arndb.de, linux-kernel@vger.kernel.org, darren@os.amperecomputing.com, yangyicong@hisilicon.com, huzhanyuan@oppo.com, lipeifeng@oppo.com, zhangshiming@oppo.com, guojian@oppo.com, realmz6@gmail.com, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Barry Song , Nadav Amit , Mel Gorman Subject: [PATCH v2 4/4] arm64: support batched/deferred tlb shootdown during page reclamation Date: Mon, 11 Jul 2022 15:46:15 +1200 Message-Id: <20220711034615.482895-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220711034615.482895-1-21cnbao@gmail.com> References: <20220711034615.482895-1-21cnbao@gmail.com> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=htIXMTHV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657511234; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kX77QQPavnlmqTPJr+cPH2hNKZO7UlqRRz965T9dDrs=; b=Os4bp1K1Xb0hCdwBd+oV2xy1hp0K+3xJLK6cFgR4ndR42fxzRtGipjTIDO6qU+KszWhb0F eyqhy/KJNDc9xYBoceX8FxuW3GOqFYRTX4DRnQmvTpARjccaVIEEpVlhWK5P9jDGW0+8jn /2PhJq+0GX+7FXl/zTusFadC1Za/zNY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657511234; a=rsa-sha256; cv=none; b=2IcT1v21XK/JveXygn1juW+/U8qZwF2ANQjMcbZHKmoOlFl7+Pkzpdso1dbB7F1mxM/OaH pkBHy6BTeGFrmgKyJn/JyFGEGmo8sRYA4J3BjHXspNeThJBacC39ZdkC/0N+hfedwTP9uk 0ZMkvnF3FYnP89NA+nTW7YRQq0JOWi4= Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=htIXMTHV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=21cnbao@gmail.com X-Stat-Signature: 1t81chq76fxzf6784iucb9ms7me1gb7q X-Rspamd-Queue-Id: 248A51C0069 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1657511233-458606 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song on x86, batched and deferred tlb shootdown has lead to 90% performance increase on tlb shootdown. on arm64, HW can do tlb shootdown without software IPI. But sync tlbi is still quite expensive. Even running a simplest program which requires swapout can prove this is true, #include #include #include #include int main() { #define SIZE (1 * 1024 * 1024) volatile unsigned char *p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); memset(p, 0x88, SIZE); for (int k = 0; k < 10000; k++) { /* swap in */ for (int i = 0; i < SIZE; i += 4096) { (void)p[i]; } /* swap out */ madvise(p, SIZE, MADV_PAGEOUT); } } Perf result on snapdragon 888 with 8 cores by using zRAM as the swap block device. ~ # perf record taskset -c 4 ./a.out [ perf record: Woken up 10 times to write data ] [ perf record: Captured and wrote 2.297 MB perf.data (60084 samples) ] ~ # perf report # To display the perf.data header info, please use --header/--header-only options. # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 60K of event 'cycles' # Event count (approx.): 35706225414 # # Overhead Command Shared Object Symbol # ........ ....... ................. ............................................................................. # 21.07% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irq 8.23% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore 6.67% a.out [kernel.kallsyms] [k] filemap_map_pages 6.16% a.out [kernel.kallsyms] [k] __zram_bvec_write 5.36% a.out [kernel.kallsyms] [k] ptep_clear_flush 3.71% a.out [kernel.kallsyms] [k] _raw_spin_lock 3.49% a.out [kernel.kallsyms] [k] memset64 1.63% a.out [kernel.kallsyms] [k] clear_page 1.42% a.out [kernel.kallsyms] [k] _raw_spin_unlock 1.26% a.out [kernel.kallsyms] [k] mod_zone_state.llvm.8525150236079521930 1.23% a.out [kernel.kallsyms] [k] xas_load 1.15% a.out [kernel.kallsyms] [k] zram_slot_lock ptep_clear_flush() takes 5.36% CPU in the micro-benchmark swapping in/out a page mapped by only one process. If the page is mapped by multiple processes, typically, like more than 100 on a phone, the overhead would be much higher as we have to run tlb flush 100 times for one single page. Plus, tlb flush overhead will increase with the number of CPU cores due to the bad scalability of tlb shootdown in HW, so those ARM64 servers should expect much higher overhead. Further perf annonate shows 95% cpu time of ptep_clear_flush is actually used by the final dsb() to wait for the completion of tlb flush. This provides us a very good chance to leverage the existing batched tlb in kernel. The minimum modification is that we only send async tlbi in the first stage and we send dsb while we have to sync in the second stage. With the above simplest micro benchmark, collapsed time to finish the program decreases around 5%. Typical collapsed time w/o patch: ~ # time taskset -c 4 ./a.out 0.21user 14.34system 0:14.69elapsed w/ patch: ~ # time taskset -c 4 ./a.out 0.22user 13.45system 0:13.80elapsed Also, Yicong Yang added the following observation. Tested with benchmark in the commit on Kunpeng920 arm64 server, observed an improvement around 12.5% with command `time ./swap_bench`. w/o w/ real 0m13.460s 0m11.771s user 0m0.248s 0m0.279s sys 0m12.039s 0m11.458s Originally it's noticed a 16.99% overhead of ptep_clear_flush() which has been eliminated by this patch: [root@localhost yang]# perf record -- ./swap_bench && perf report [...] 16.99% swap_bench [kernel.kallsyms] [k] ptep_clear_flush Cc: Jonathan Corbet Cc: Nadav Amit Cc: Mel Gorman Tested-by: Yicong Yang Signed-off-by: Barry Song --- .../features/vm/TLB/arch-support.txt | 2 +- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/tlbbatch.h | 12 ++++++++++ arch/arm64/include/asm/tlbflush.h | 23 +++++++++++++++++-- 4 files changed, 35 insertions(+), 3 deletions(-) create mode 100644 arch/arm64/include/asm/tlbbatch.h diff --git a/Documentation/features/vm/TLB/arch-support.txt b/Documentation/features/vm/TLB/arch-support.txt index 1c009312b9c1..2caf815d7c6c 100644 --- a/Documentation/features/vm/TLB/arch-support.txt +++ b/Documentation/features/vm/TLB/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | TODO | + | arm64: | ok | | csky: | TODO | | hexagon: | TODO | | ia64: | TODO | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 1652a9800ebe..e94913a0b040 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -93,6 +93,7 @@ config ARM64 select ARCH_SUPPORTS_INT128 if CC_HAS_INT128 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK + select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/include/asm/tlbbatch.h b/arch/arm64/include/asm/tlbbatch.h new file mode 100644 index 000000000000..fedb0b87b8db --- /dev/null +++ b/arch/arm64/include/asm/tlbbatch.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ARCH_ARM64_TLBBATCH_H +#define _ARCH_ARM64_TLBBATCH_H + +struct arch_tlbflush_unmap_batch { + /* + * For arm64, HW can do tlb shootdown, so we don't + * need to record cpumask for sending IPI + */ +}; + +#endif /* _ARCH_ARM64_TLBBATCH_H */ diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25..10364cf8451d 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -254,17 +254,24 @@ static inline void flush_tlb_mm(struct mm_struct *mm) dsb(ish); } -static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, + +static inline void __flush_tlb_page_nosync(struct mm_struct *mm, unsigned long uaddr) { unsigned long addr; dsb(ishst); - addr = __TLBI_VADDR(uaddr, ASID(vma->vm_mm)); + addr = __TLBI_VADDR(uaddr, ASID(mm)); __tlbi(vale1is, addr); __tlbi_user(vale1is, addr); } +static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, + unsigned long uaddr) +{ + return __flush_tlb_page_nosync(vma->vm_mm, uaddr); +} + static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) { @@ -272,6 +279,18 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, dsb(ish); } +static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, + unsigned long uaddr) +{ + __flush_tlb_page_nosync(mm, uaddr); +} + +static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) +{ + dsb(ish); +} + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement.