From patchwork Mon Jul 17 13:10:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yicong Yang X-Patchwork-Id: 13315653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEA37C0015E for ; Mon, 17 Jul 2023 13:12:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 632AB8D0002; Mon, 17 Jul 2023 09:12:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BC566B0074; Mon, 17 Jul 2023 09:12:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45BB98D0002; Mon, 17 Jul 2023 09:12:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3771B6B0072 for ; Mon, 17 Jul 2023 09:12:16 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 07B501402C5 for ; Mon, 17 Jul 2023 13:12:16 +0000 (UTC) X-FDA: 81021142272.19.A876244 Received: from szxga01-in.huawei.com (szxga01-in.huawei.com [45.249.212.187]) by imf07.hostedemail.com (Postfix) with ESMTP id 11A2B40018 for ; Mon, 17 Jul 2023 13:12:12 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf07.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=yangyicong@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689599533; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tQDXL5G5kaAGoo/7M++qyEMAn71EHHw+DYe+eZ1T3qc=; b=NmblVhgxh6Z8XZsOSXN5F/Y1Nzsop2DKQsBqitYdTPFxmIn44q/xoGV+g6BkKdiW5iqe0u gzutwYbu2fSWszElhAoBa/+CzifjIkK+K2yZhFVx38vS/X9WANP48aurOnN0wTIy551mGy eU0SQkp8n6Ws6VmtBowCk1w46jTrMdY= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf07.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.187 as permitted sender) smtp.mailfrom=yangyicong@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689599533; a=rsa-sha256; cv=none; b=OyULYvr3tsjs56RM4TepB3xFqIaYFSjCSa7RM/xiJ48mK+AHCToMrOAlVWA4+05g69tXip QezPTLq1ZDMJJQrpNDe9A0v9sEspQUrk1JS8OF1OSvfCuBClGq/XxNcPlsgtvI4Pw9/skm Mx/IBI09iuXXE9zEFHiEF9MqilWKVUw= Received: from canpemm500009.china.huawei.com (unknown [172.30.72.55]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4R4Mq35PpqztRVP; Mon, 17 Jul 2023 21:09:03 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by canpemm500009.china.huawei.com (7.192.105.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 17 Jul 2023 21:12:08 +0800 From: Yicong Yang To: , , , , , , , , , CC: , , , , , , , , , , , , , , , , , Barry Song <21cnbao@gmail.com>, , , , , Anshuman Khandual , Barry Song Subject: [PATCH v11 1/4] mm/tlbbatch: Introduce arch_tlbbatch_should_defer() Date: Mon, 17 Jul 2023 21:10:01 +0800 Message-ID: <20230717131004.12662-2-yangyicong@huawei.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20230717131004.12662-1-yangyicong@huawei.com> References: <20230717131004.12662-1-yangyicong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To canpemm500009.china.huawei.com (7.192.105.203) X-CFilter-Loop: Reflected X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 11A2B40018 X-Stat-Signature: rm8kg8st69ciqrgjoma1i8yio34jtu4j X-Rspam-User: X-HE-Tag: 1689599532-906787 X-HE-Meta: U2FsdGVkX1+2Wa2/XQKQUD7e8rlHMIJ3nwF67GrO0Rey2CA4DGvx7KV5+mFfaTe2klEgHdlw4pyZU88VN2lM/ygSkPUlsXEdCBHpC+hILufrEKXWVt3KT5buYYqOuhm82zgAOkUn3hHwXjmVaF7bK8pjqTcAhM37PkMsqcAp8lEOH88ZPA/D6/rSfeHA8+Vw/HiwBh68UjO4tX3t6rxk7wCrYdDrtsmWeD7mkRzDGYIMFu/0rCNdi+KxdcxSKkq6IokjET5AtCTcWsEI1ajlZYL6R7arhlbUUDzWZjRHZEYhBx439wKXWr5zGTWc2UQi9hHFg8inKwCN1ZwAslRVzzjVLppqjEFKTvdDH59RwTSSCgf9uuJZGEew+IY/eVRjurTfbKPtGXepCmtTYR9+K0x4vcsxvfkEB6JBun+fzj+9bMrK6AfCjL5w1SUVmiOy+QhNKeOKWNcgzX256wP/a6lD+xMWRT0SmdmRmUDFYAKRwLMmJW6T+tziC1HDK0WyApa64c750qhY8etB6jzsItjg6b1eCaicNRZu+BB3uGK7J7g874f1IcrlB8GNV7kvIsW91nEoibcehbsFxPUiu9+6FcmVzqbqpUizIVNZx118i7WwCzpNI3PMhsZsT4AwptN/mEGH24nhb9Oz+cdl9os3ti2G3PgHZl5jK272yZDOtIjNxrL6EJ6gDCkuPk6/9A3cWfcce5hEmDjIFti4nA07Hl6DIc6XBRdLLDGPRMRiQb2AKWaF+Kg47yOz1HrQgACvIRCGJmHzEjGUDptEBh7pM5KzXGpsaMqvPRF7XX0ae1yNDABuS/Wn+6LXk26VyjbT1vT+bpzHMBfHCR4o+bGQdYseCE7U39ciUTyo44GSK3aVHmjtRxxkdKFKAk6z9vwDFqu1jAgV+DKDMgwY9zuSwywQV0dDFPnxEnfEidpgksxt3nnjhJyeWsN0Zp5F3YKB5fOQ7RB+8gGsCLl Eohudxvp NTWWeNFksaoHuG66IhRAl4MpQZwSjHKt4G0MG9Jfy3B0I5hxMfazuy2KsgjcVcKSj1//1VitGDEu4YhHKBLlSsU+WWrH2ZUrLzB3vU9EUmECgG1qs4fNwAUTKXDOndGumrqxPMKm+uG8fzMrPxlXin2ERFaN/vcwyQ6Ef3mt7v/5ONWmNKgyMNF2cIvwozdZUYJkbndN/u9Q7LMvgVcNRLUSAQVj5EkvGNgYN2MNmBI8Rsi3gwkg9hXqjXKYYZc/+lWSsjuq++nfDW6DANCD2nvK9s/Xem4hhu9cKfsR/R/fvzT1zS13m0sEVt0He6ifRaSqF67N7iwspKCqXnvtSrlYlcUMYNtgvdQ7OAKcJMt48JIzILLlPNCqkLLnQpZF2iY14A5hbFpb4CRsjiqfvfob2WJfN3odWhgYg5YsRhA9WnAjmhAoUWDOg1JPwH3FLJ398zffaPxgOQi0MgJk8Zm8fn2wyo7wHrGac X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Anshuman Khandual The entire scheme of deferred TLB flush in reclaim path rests on the fact that the cost to refill TLB entries is less than flushing out individual entries by sending IPI to remote CPUs. But architecture can have different ways to evaluate that. Hence apart from checking TTU_BATCH_FLUSH in the TTU flags, rest of the decision should be architecture specific. Signed-off-by: Anshuman Khandual [https://lore.kernel.org/linuxppc-dev/20171101101735.2318-2-khandual@linux.vnet.ibm.com/] Signed-off-by: Yicong Yang [Rebase and fix incorrect return value type] Reviewed-by: Kefeng Wang Reviewed-by: Anshuman Khandual Reviewed-by: Barry Song Reviewed-by: Xin Hao Tested-by: Punit Agrawal Reviewed-by: Catalin Marinas --- arch/x86/include/asm/tlbflush.h | 12 ++++++++++++ mm/rmap.c | 9 +-------- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 80450e1d5385..cf2a1de5d388 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -253,6 +253,18 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false); } +static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm) +{ + bool should_defer = false; + + /* If remote CPUs need to be flushed then defer batch the flush */ + if (cpumask_any_but(mm_cpumask(mm), get_cpu()) < nr_cpu_ids) + should_defer = true; + put_cpu(); + + return should_defer; +} + static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) { /* diff --git a/mm/rmap.c b/mm/rmap.c index 0c0d8857dfce..6480e526c154 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -688,17 +688,10 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval) */ static bool should_defer_flush(struct mm_struct *mm, enum ttu_flags flags) { - bool should_defer = false; - if (!(flags & TTU_BATCH_FLUSH)) return false; - /* If remote CPUs need to be flushed then defer batch the flush */ - if (cpumask_any_but(mm_cpumask(mm), get_cpu()) < nr_cpu_ids) - should_defer = true; - put_cpu(); - - return should_defer; + return arch_tlbbatch_should_defer(mm); } /* From patchwork Mon Jul 17 13:10:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yicong Yang X-Patchwork-Id: 13315654 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D738C001DC for ; Mon, 17 Jul 2023 13:12:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 215118D0003; Mon, 17 Jul 2023 09:12:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C4916B0074; Mon, 17 Jul 2023 09:12:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F323C8D0003; Mon, 17 Jul 2023 09:12:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E2CF66B0072 for ; Mon, 17 Jul 2023 09:12:17 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A1A2BA030D for ; Mon, 17 Jul 2023 13:12:17 +0000 (UTC) X-FDA: 81021142314.27.EE264D6 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by imf14.hostedemail.com (Postfix) with ESMTP id 919A9100026 for ; Mon, 17 Jul 2023 13:12:13 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=yangyicong@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689599534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sUBByuZXbpyKrc9wDQ9lkzXHe8yqE1f/9cc7ANd+/4c=; b=MwrewPdASXa/soy2+D//zgb7u8EYasgfCMB+Ddy5oyeIsbk8dmfGeO8rv1NDPkWaNq64zV DM3YibVMON1gLOb/sOf5mpY9nosIwcNzliwatUdLz9m1dV6NF4qWYGH7oLeaUgfmnf404Z 0IxjtxjkHTSbXcfp30avI/Z/hC1znTc= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=yangyicong@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689599534; a=rsa-sha256; cv=none; b=UU3QJIOxnl2G1iXjr4oasM8Lkw4eBObGFYmV97erD8CE7bWI4uBiroxPHALm8nPt5LsmR4 m8Hz74b0swuxIDGtrBumKBsXHs7xLfQQNT30iSct8NJqZPK1MktNT3WTPTOJVV5OCbh7Al N7SIqnrtiy5X89ozhlF5HxBeNo8Hs00= Received: from canpemm500009.china.huawei.com (unknown [172.30.72.54]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4R4Mqr0bWqzLngj; Mon, 17 Jul 2023 21:09:44 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by canpemm500009.china.huawei.com (7.192.105.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 17 Jul 2023 21:12:09 +0800 From: Yicong Yang To: , , , , , , , , , CC: , , , , , , , , , , , , , , , , , Barry Song <21cnbao@gmail.com>, , , , , Barry Song , Nadav Amit , Mel Gorman Subject: [PATCH v11 2/4] mm/tlbbatch: Rename and extend some functions Date: Mon, 17 Jul 2023 21:10:02 +0800 Message-ID: <20230717131004.12662-3-yangyicong@huawei.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20230717131004.12662-1-yangyicong@huawei.com> References: <20230717131004.12662-1-yangyicong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To canpemm500009.china.huawei.com (7.192.105.203) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 919A9100026 X-Rspam-User: X-Stat-Signature: gynsr45ti5j47scaqr93dhu45nn89gxf X-Rspamd-Server: rspam01 X-HE-Tag: 1689599533-436682 X-HE-Meta: U2FsdGVkX1/PvRcbX/Coz4hltxK1nfPiHPYI+R1FLt5XqT34o6rY0qpG87jlNmr1ZMT2jX7udFPaVluwiBLqzH05RHkTOdhR51UPlEFWautJe5RmpDFGt0wt7u6/NorYGIKxcDxtAKDFOGuR7uotGAPst2Vew911pVC5AhhXEK3y5OYsl12EN2jReioKnkS1j7gc5UYutKXNcf0aHlvRoKtCRcZTFAgv9C47+JEDER19/wVF5mLjyYW/lW1tXeNWb131XoUOtUzn8N1fILQB9Ji7mF5Bp36c3UBWmWsjzjspl7GwJLshC1uTlgBrdfC3a/q2KqQG3LqroUMnyUxprQNIW5NS3eh7vw2gr6gYIB/Tcfu0PJFl4EbI2X7hJb9r0dpvor6ySnkegTWqYTrm/dTNgjda2q9AB63COMX/GIRw/3QDOcoPFFO8D7r43dRUBaddEE100oJcsmHobnXgQqBry/FTFWDj6VBAW3ZmPUrNJmODCA4fbcTY5teEH0MLrSiIUeBL8lOTDNXu++KAKo9DX1cvYQhDCg2BkiddOPtfABuRfAbGjUyP5Gl41c12QzO1PxH6tI/MX+KbU9/2XVktdJKVW5PjYYQ3xBhebxVPVbXVX5q4mN/9iCF93gXxZnP7dyMlzCAfLqw9ntF8wN985lmy+2pfOX2wfTeVkwosKzHF3FmV4glYipurv5/SOi54eQ6XOVssdTH47GcuVvKHcFbRwfVZJt47DqazTsn0lXwGiqwq26kmuxZw1N3czzoDdpg5EfcXsgYszc6aUJ9aZid2it921wIoPQ9QhqlqxlA48IG9quvkFkiD9nEtC/wV1aOXKDpQge7rcYpBBceXcJafp1HmPbSA3T45S/DyD9LeFUfXFBOn9Vet6zcEs+NboFNcxfJtR0Z7gjBFgRjpLV4+0Be4uRqSeFTDKs9eJq19o8fcU+t+sYkZAAyqPw/wjA9L7RrKSdPUfyQ DSke2a2U AF1yRW71GHVz9psu3tWBZHQGWXPlPxv1pskMrZzno7J9EH5L56iNFZwP7pP0hazxQ5LGUBw5Ujm1WhrqJ8vBv6uXhiXTq8X+L4+Wte25z+wOKzixdmzVvGnuacoPEOyUuu8iODe2dd/SgZYRUpRoi0VFVnKZJoqVTV6HKNUx2iZ/dyAaZoey0FS3ROAvJ3pt+flKocivo7D8o/psHreC7v+Pm4mi3o5VtlgYSnukl57PZb38uuiIJnr38F+CxO//p3SH236VQT0bsDCAoPAkpiM/eerhzS4K+APr870sU3exj0ObagdTCe1TEQ/wRg3fjrWJXRRkzOHIrCcTUQKhbbUTi0SA89F7cKjHEURi6CX6EWzx2PZAhVGaHNuelgRFRyGOpWoTKndocf5RwP95RsdBUNSOzgTzMaP4k0xMhjw3HvRM00cYWZ3AqG+IXCBYf6ax8nRBUO5KkcroRjK+r7gRcXHYMpgipp9qW9JvjggPGhE6tHSn2Xps2SA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song This patch does some preparation works to extend batched TLB flush to arm64. Including: - Extend set_tlb_ubc_flush_pending() and arch_tlbbatch_add_mm() to accept an additional argument for address, architectures like arm64 may need this for tlbi. - Rename arch_tlbbatch_add_mm() to arch_tlbbatch_add_pending() to match its current function since we don't need to handle mm on architectures like arm64 and add_mm is not proper, add_pending will make sense to both as on x86 we're pending the TLB flush operations while on arm64 we're pending the synchronize operations. This intends no functional changes on x86. Cc: Anshuman Khandual Cc: Jonathan Corbet Cc: Nadav Amit Cc: Mel Gorman Tested-by: Yicong Yang Tested-by: Xin Hao Tested-by: Punit Agrawal Signed-off-by: Barry Song Signed-off-by: Yicong Yang Reviewed-by: Kefeng Wang Reviewed-by: Xin Hao Reviewed-by: Anshuman Khandual Reviewed-by: Catalin Marinas --- arch/x86/include/asm/tlbflush.h | 5 +++-- include/linux/mm_types_task.h | 4 ++-- mm/rmap.c | 12 +++++++----- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index cf2a1de5d388..1c7d3a36e16c 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -276,8 +276,9 @@ static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) return atomic64_inc_return(&mm->context.tlb_gen); } -static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, - struct mm_struct *mm) +static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, + unsigned long uaddr) { inc_mm_tlb_gen(mm); cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); diff --git a/include/linux/mm_types_task.h b/include/linux/mm_types_task.h index 5414b5c6a103..aa44fff8bb9d 100644 --- a/include/linux/mm_types_task.h +++ b/include/linux/mm_types_task.h @@ -52,8 +52,8 @@ struct tlbflush_unmap_batch { #ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH /* * The arch code makes the following promise: generic code can modify a - * PTE, then call arch_tlbbatch_add_mm() (which internally provides all - * needed barriers), then call arch_tlbbatch_flush(), and the entries + * PTE, then call arch_tlbbatch_add_pending() (which internally provides + * all needed barriers), then call arch_tlbbatch_flush(), and the entries * will be flushed on all CPUs by the time that arch_tlbbatch_flush() * returns. */ diff --git a/mm/rmap.c b/mm/rmap.c index 6480e526c154..9699c6011b0e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -642,7 +642,8 @@ void try_to_unmap_flush_dirty(void) #define TLB_FLUSH_BATCH_PENDING_LARGE \ (TLB_FLUSH_BATCH_PENDING_MASK / 2) -static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval) +static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, + unsigned long uaddr) { struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; int batch; @@ -651,7 +652,7 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval) if (!pte_accessible(mm, pteval)) return; - arch_tlbbatch_add_mm(&tlb_ubc->arch, mm); + arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; /* @@ -726,7 +727,8 @@ void flush_tlb_batched_pending(struct mm_struct *mm) } } #else -static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval) +static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, + unsigned long uaddr) { } @@ -1579,7 +1581,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pteval); + set_tlb_ubc_flush_pending(mm, pteval, address); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } @@ -1962,7 +1964,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pteval); + set_tlb_ubc_flush_pending(mm, pteval, address); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } From patchwork Mon Jul 17 13:10:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yicong Yang X-Patchwork-Id: 13315656 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F66AC001B0 for ; Mon, 17 Jul 2023 13:12:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D371E6B0072; Mon, 17 Jul 2023 09:12:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0E0E8D0006; Mon, 17 Jul 2023 09:12:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD5D26B0075; Mon, 17 Jul 2023 09:12:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id AEAEE6B0072 for ; Mon, 17 Jul 2023 09:12:21 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 59EE91A02F2 for ; Mon, 17 Jul 2023 13:12:21 +0000 (UTC) X-FDA: 81021142482.12.75186E8 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf02.hostedemail.com (Postfix) with ESMTP id 953C38001B for ; Mon, 17 Jul 2023 13:12:18 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=yangyicong@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689599539; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fhZ+V4S52X/kXuQV51PeKIPPTeN+0eX+HbKE+khnxNc=; b=rWPIuNPoMM4UJUPu0lZTg1p20KQ4ZRK9eZGQVoyq78xPZ53z8Qz+HUqrwz0AZMyeyKIvV8 4JwMRnsqfXHQizgX+Ao1AY2nOiyTfbPFMIRzNNB5l5qbdqCeCnLWgVBF9Huduy3XYeg8Pk VRH8z7NkfEbAaeTP3mZENGIRPoHUHJ8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689599539; a=rsa-sha256; cv=none; b=VbHRf5PG0EUn+sRkmDFsOGI9uHehtV8ExR+eNs/qvukpdE60OLNJQm3vZ7B+n6G2Ft9D76 wp6Eefoo7HA4HrEijyVcLmjQf09UiMT8NQJFCqIhx2wjdTWI2ZkB3mqHUSkMs8rAVomJJ+ vDt0nh3lnhgn1RiPj6ZTXnPNYq/gvHw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=yangyicong@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from canpemm500009.china.huawei.com (unknown [172.30.72.55]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4R4Mps5c95zNmN0; Mon, 17 Jul 2023 21:08:53 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by canpemm500009.china.huawei.com (7.192.105.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 17 Jul 2023 21:12:12 +0800 From: Yicong Yang To: , , , , , , , , , CC: , , , , , , , , , , , , , , , , , Barry Song <21cnbao@gmail.com>, , , , Subject: [PATCH v11 3/4] mm/tlbbatch: Introduce arch_flush_tlb_batched_pending() Date: Mon, 17 Jul 2023 21:10:03 +0800 Message-ID: <20230717131004.12662-4-yangyicong@huawei.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20230717131004.12662-1-yangyicong@huawei.com> References: <20230717131004.12662-1-yangyicong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To canpemm500009.china.huawei.com (7.192.105.203) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 953C38001B X-Rspam-User: X-Stat-Signature: wntp5fschjtumatgfwfr9pnhiqbqwbd9 X-Rspamd-Server: rspam03 X-HE-Tag: 1689599538-788147 X-HE-Meta: U2FsdGVkX19cWDgp9hp196Gh0+FiuC0Js3+7rh6D2+ZrhurXHFojCuOTn4772Z7FfgLBBnFvscuTNkjb86NGHiKcip/U/ZRg/cGanrG2wHBn/qg9Yo2Hj7BwqSSOckTLQA4atCfKU9bxMU29YjUwwWhK+NDHfdvNhw4vqGOCUfWjDK6vRapCGOHMHkalZWUJKxf5ac8rPMe/EhHmWca0o0VgRdJ9tq5saAtauR5sI1DXcpcJ6MfdETauAHi/2CZKUCZMXnxiC3s+oKuldsv0CuX5kkiuz6LO5ZOd3oE4NK1fIwElJtlQWPauOSqEVWw3euyzke8IbjlLeXBeIXq8dPx7z3RR6ZFyWCnliGtgqlVPCLLswfsxdC3hl3EpUfezivo/TJAGmGXjBCxOlHdqYHHO5jibYFbl7Iq5Ia6DFgRUJvddPnZ63LZwL78KxYdlLMOcsegDJUpNJb6WmHDQ7mEaXdwvBQ64q4MWLqBxg3xdnPzINeVtfIJrmO9KosyciT1IVFtlambRDE68RL8NbmpOLBb19T9jMISYWQOWca6mLXhxbP3zXUHppmpzebVG6LU7N8IwHcz7/QGfkIcVaUXuGdplR6WdXR5XwekF1E80wkyE2UJ+pdgDHplmR7pmNoU1SFA2577czEpXQNIGwsEeUAWFPdKxRgiuhrxqHC6JKhs8bQQlgNeGGEPoM6WCJNo3+R4CX56Uk/XOSKFNB0LMcSjiljYMZCv28JQopfXYPiuJX5HvastK0yxsgTkHGRVnKWG318Qoa0ZllJhiV0fPAv3lj+KlTgZ/pGz/QZ/QhPoDtJndo7NxjzOrkf1zn3BmWXqXNsRZ42eDaiUWaagEF21iRNUuH6IemfIyORuVdJVGddCdd4WFFOK5vY8+7yU4cGMYeCmo+2Jq3YnpjU6hmn12ru93uEja/mfl9WhUMXowHIaMovOctyMr5BtEb/Ev4aBzEdKCvuNxDj9 SDcmOUVj 8iABh9qKNDooRcpfx5nruWH9tP5O4/ZcIzzq06bGRnbViK3ZX9n8yYJSxhyczlmkkokBTanGfzWMiDvegufoAWowYto+7isAXZ3d9tkeO1Yxo8pOZp8xSXmat9Yke9ElsHsEdDSS999qKBVndO3/wo5dG7upynaTIUFtDGg1neV2uCfolqaCJk5B59jGonCbDrIhxx/zsd+JZ1k8dYidr1Yha75kYFMNbRKIRTxOmvTMDEXo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yicong Yang Currently we'll flush the mm in flush_tlb_batched_pending() to avoid race between reclaim unmaps pages by batched TLB flush and mprotect/munmap/etc. Other architectures like arm64 may only need a synchronization barrier(dsb) here rather than a full mm flush. So add arch_flush_tlb_batched_pending() to allow an arch-specific implementation here. This intends no functional changes on x86 since still a full mm flush for x86. Signed-off-by: Yicong Yang Reviewed-by: Catalin Marinas --- arch/x86/include/asm/tlbflush.h | 5 +++++ mm/rmap.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 1c7d3a36e16c..837e4a50281a 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -284,6 +284,11 @@ static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *b cpumask_or(&batch->cpumask, &batch->cpumask, mm_cpumask(mm)); } +static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) +{ + flush_tlb_mm(mm); +} + extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); static inline bool pte_flags_need_flush(unsigned long oldflags, diff --git a/mm/rmap.c b/mm/rmap.c index 9699c6011b0e..3a16c91be7e2 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -717,7 +717,7 @@ void flush_tlb_batched_pending(struct mm_struct *mm) int flushed = batch >> TLB_FLUSH_BATCH_FLUSHED_SHIFT; if (pending != flushed) { - flush_tlb_mm(mm); + arch_flush_tlb_batched_pending(mm); /* * If the new TLB flushing is pending during flushing, leave * mm->tlb_flush_batched as is, to avoid losing flushing. From patchwork Mon Jul 17 13:10:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yicong Yang X-Patchwork-Id: 13315655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03BA1EB64DC for ; Mon, 17 Jul 2023 13:12:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A3738D0005; Mon, 17 Jul 2023 09:12:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 805B56B0074; Mon, 17 Jul 2023 09:12:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F44E8D0005; Mon, 17 Jul 2023 09:12:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 62ABA6B0072 for ; Mon, 17 Jul 2023 09:12:21 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 03E9C160311 for ; Mon, 17 Jul 2023 13:12:20 +0000 (UTC) X-FDA: 81021142482.21.E3BB5B2 Received: from szxga02-in.huawei.com (szxga02-in.huawei.com [45.249.212.188]) by imf19.hostedemail.com (Postfix) with ESMTP id CF6331A0021 for ; Mon, 17 Jul 2023 13:12:17 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf19.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=yangyicong@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689599538; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Pfyo8UxbV8aGczldhxgYNeAHUd6Qhl2hUsMsId8Wlfs=; b=6Qklq6biZsH7iNImfl0qbh2B1bIKDrMYZWCWrmR/bjxCaWTQeEkvBWdBHwncyjeyQmtuUO sQd9wHYpWcNE4egF+09O0Nny6+pIr7AD027MAcBiiUFsJ5Sdry90wmEDPdD0g+cbtik3Jh GgOlRFpKYmvCVqirMp2LWqp8TBtMv0c= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf19.hostedemail.com: domain of yangyicong@huawei.com designates 45.249.212.188 as permitted sender) smtp.mailfrom=yangyicong@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689599538; a=rsa-sha256; cv=none; b=ByVubvF/B2Hxv79VAI9cF6T2i+yFh0bkLPWYILkFE6DGp/F39c3rNYvv5QIaPXLnWAs5JS vZu1eDEn1hofFCIU6vrrebcMrF0xXxPVy/LQVffNyEfut1ayOuWNIGEn0+N5nN36R9V2MZ Gv3nEzJmzn/pPaCbZBZ1K1I9EGS1MYw= Received: from canpemm500009.china.huawei.com (unknown [172.30.72.54]) by szxga02-in.huawei.com (SkyGuard) with ESMTP id 4R4Mpt6HzczNmSV; Mon, 17 Jul 2023 21:08:54 +0800 (CST) Received: from localhost.localdomain (10.50.163.32) by canpemm500009.china.huawei.com (7.192.105.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Mon, 17 Jul 2023 21:12:13 +0800 From: Yicong Yang To: , , , , , , , , , CC: , , , , , , , , , , , , , , , , , Barry Song <21cnbao@gmail.com>, , , , , Barry Song , Nadav Amit , Mel Gorman Subject: [PATCH v11 4/4] arm64: support batched/deferred tlb shootdown during page reclamation/migration Date: Mon, 17 Jul 2023 21:10:04 +0800 Message-ID: <20230717131004.12662-5-yangyicong@huawei.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20230717131004.12662-1-yangyicong@huawei.com> References: <20230717131004.12662-1-yangyicong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.50.163.32] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To canpemm500009.china.huawei.com (7.192.105.203) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: CF6331A0021 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: fpbidb7rgasne6y4z95zr5nxpih5ioyc X-HE-Tag: 1689599537-660010 X-HE-Meta: U2FsdGVkX1+WKJ10G5l/0X18Jzy3m+PAD5kVfNwTmSZnMvrgzzYuGYkuC05ONoYwYja0anzwTMy0HXMdARlV+s70fZ2Ecdyi7fRH3E37S65RyeGIdczlzaXs9uF75tCxycnxwpO/5nwWstZ4Bn9nl+uiO3puBPUP9zfYf+QAi1XjQkqvv6gHu+UQOrJVmjqpA86Ym27lcvV1e65w5dJ5vNmZ5Vvp0Sj1giVpOBoLtnOWQ9bj90L3emFFVIwZM9gqcfuMiBrCqIqvL5V3BQCfaRHcZCHjVbqnt72hpbMptdmTELHd8F6bQpkBhfeRl5ngAfQTgPryZ9qXEv5wDOUGfG1uMtERMEx7MUXALYEfT1FP4sYTCAD5qqRep1gDnA0xGhaNlkFhHtP4P0LSumP3Mc6++viGkNOPfyS/jUwLU4he0J6K+5KjPhlPyLJbz1EOFHtIGD//AyfPP/N9lZZRPp3gMtpMxSHsak3GR83Fzkfd5tHb2Uo+/kti+TGyvsYahIyzB1YwIEDcXtGYEHW0/ga6Kdt0CL8B1wRab2GNTiFEZQdiLa5xJDjvGvHEMOEE/ZdGHav57jxTNvp8W2tflqy27pNnjohq7+o5VgsqRX8IUeEbEQne6zKkJDHypvWmks5yNF0WueVbRNDH5hI8Iggw/zB5j07iSAt67jLNMJ/EGItVvgn31Wn8PXGCVNhefUxOE/j/Y7y3ewgv3NdYappADGhczUb6sSKHurMD09P6+ZYsdVWuHDHwxMQCIW9LhfyvA0zwOft9zqQvSszDjJtd5oUuuAcLv5a/t26rgeDyT3X+D6v4RExX43XDlOnu4LKcMYoqrsUTGV5HaqOGQFUO6/S79OmtXvCqz53O0wAEMHT1gwkSa1mhYA4inua85vwI/ewz/Tf/GmIZ7boRLElIT8QWIsVzHl84J++xlKvxjado9HIUMnklMFsKKieOpGLPHbJfSv+8IX99Oeo DvMQWl+D HpSs9diR2NB4zhJkxN3ZGH5Njq48bn9V6WUW6Cc1zetBfU224hi3C/Yg1ca9/80LJkfQV1RfKVw/uKYvqPyDPKiKi8HXXUKHl4tj2C78P47WXRE30hBt91wjrNqP7IsKSqoDJT93dRWYosZ8y9yZcE9ylJ697tAJDkXy6CBck++alvmP33jfNan1ZDUQkllI29KZYQSbI0Rcv4zKiUjYaCOuORDmNFEUoyBCKJi7lKjjaIcWNQ+Muikw0gRX76FB2XGEZI52xSmH8HOew1XM6nzTQgD8LHlWJ+MKTxHUXHcrioWyAUu02ym4tw2vj7hWDEUPUAeAPI19cthOmkkMKKpskT+lbuORYy+fRb0V/xRdpMZTAqOj5QkSzG/oZeGhFBjE2wn7JFReBCfcXRMmawDlUOCiODKC12AYVkVU3JtOOt42BD+14xgVxqBlVv8cGVAkXE4NtNTBrSV5v4DemdJRfwJgMmhbhM2cNAlBFYGRl3apnlHVJ/ehWyw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Barry Song on x86, batched and deferred tlb shootdown has lead to 90% performance increase on tlb shootdown. on arm64, HW can do tlb shootdown without software IPI. But sync tlbi is still quite expensive. Even running a simplest program which requires swapout can prove this is true, #include #include #include #include int main() { #define SIZE (1 * 1024 * 1024) volatile unsigned char *p = mmap(NULL, SIZE, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); memset(p, 0x88, SIZE); for (int k = 0; k < 10000; k++) { /* swap in */ for (int i = 0; i < SIZE; i += 4096) { (void)p[i]; } /* swap out */ madvise(p, SIZE, MADV_PAGEOUT); } } Perf result on snapdragon 888 with 8 cores by using zRAM as the swap block device. ~ # perf record taskset -c 4 ./a.out [ perf record: Woken up 10 times to write data ] [ perf record: Captured and wrote 2.297 MB perf.data (60084 samples) ] ~ # perf report # To display the perf.data header info, please use --header/--header-only options. # To display the perf.data header info, please use --header/--header-only options. # # # Total Lost Samples: 0 # # Samples: 60K of event 'cycles' # Event count (approx.): 35706225414 # # Overhead Command Shared Object Symbol # ........ ....... ................. ...... # 21.07% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irq 8.23% a.out [kernel.kallsyms] [k] _raw_spin_unlock_irqrestore 6.67% a.out [kernel.kallsyms] [k] filemap_map_pages 6.16% a.out [kernel.kallsyms] [k] __zram_bvec_write 5.36% a.out [kernel.kallsyms] [k] ptep_clear_flush 3.71% a.out [kernel.kallsyms] [k] _raw_spin_lock 3.49% a.out [kernel.kallsyms] [k] memset64 1.63% a.out [kernel.kallsyms] [k] clear_page 1.42% a.out [kernel.kallsyms] [k] _raw_spin_unlock 1.26% a.out [kernel.kallsyms] [k] mod_zone_state.llvm.8525150236079521930 1.23% a.out [kernel.kallsyms] [k] xas_load 1.15% a.out [kernel.kallsyms] [k] zram_slot_lock ptep_clear_flush() takes 5.36% CPU in the micro-benchmark swapping in/out a page mapped by only one process. If the page is mapped by multiple processes, typically, like more than 100 on a phone, the overhead would be much higher as we have to run tlb flush 100 times for one single page. Plus, tlb flush overhead will increase with the number of CPU cores due to the bad scalability of tlb shootdown in HW, so those ARM64 servers should expect much higher overhead. Further perf annonate shows 95% cpu time of ptep_clear_flush is actually used by the final dsb() to wait for the completion of tlb flush. This provides us a very good chance to leverage the existing batched tlb in kernel. The minimum modification is that we only send async tlbi in the first stage and we send dsb while we have to sync in the second stage. With the above simplest micro benchmark, collapsed time to finish the program decreases around 5%. Typical collapsed time w/o patch: ~ # time taskset -c 4 ./a.out 0.21user 14.34system 0:14.69elapsed w/ patch: ~ # time taskset -c 4 ./a.out 0.22user 13.45system 0:13.80elapsed Also tested with benchmark in the commit on Kunpeng920 arm64 server and observed an improvement around 12.5% with command `time ./swap_bench`. w/o w/ real 0m13.460s 0m11.771s user 0m0.248s 0m0.279s sys 0m12.039s 0m11.458s Originally it's noticed a 16.99% overhead of ptep_clear_flush() which has been eliminated by this patch: [root@localhost yang]# perf record -- ./swap_bench && perf report [...] 16.99% swap_bench [kernel.kallsyms] [k] ptep_clear_flush It is tested on 4,8,128 CPU platforms and shows to be beneficial on large systems but may not have improvement on small systems like on a 4 CPU platform. Also this patch improve the performance of page migration. Using pmbench and tries to migrate the pages of pmbench between node 0 and node 1 for 100 times for 1G memory, this patch decrease the time used around 20% (prev 18.338318910 sec after 13.981866350 sec) and saved the time used by ptep_clear_flush(). Cc: Anshuman Khandual Cc: Jonathan Corbet Cc: Nadav Amit Cc: Mel Gorman Tested-by: Yicong Yang Tested-by: Xin Hao Tested-by: Punit Agrawal Signed-off-by: Barry Song Signed-off-by: Yicong Yang Reviewed-by: Kefeng Wang Reviewed-by: Xin Hao Reviewed-by: Anshuman Khandual Reviewed-by: Catalin Marinas --- .../features/vm/TLB/arch-support.txt | 2 +- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/tlbbatch.h | 12 +++++ arch/arm64/include/asm/tlbflush.h | 44 +++++++++++++++++-- 4 files changed, 55 insertions(+), 4 deletions(-) create mode 100644 arch/arm64/include/asm/tlbbatch.h diff --git a/Documentation/features/vm/TLB/arch-support.txt b/Documentation/features/vm/TLB/arch-support.txt index 7f049c251a79..76208db88f3b 100644 --- a/Documentation/features/vm/TLB/arch-support.txt +++ b/Documentation/features/vm/TLB/arch-support.txt @@ -9,7 +9,7 @@ | alpha: | TODO | | arc: | TODO | | arm: | TODO | - | arm64: | N/A | + | arm64: | ok | | csky: | TODO | | hexagon: | TODO | | ia64: | TODO | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7856c3a3e35a..b1573257a4d6 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -96,6 +96,7 @@ config ARM64 select ARCH_SUPPORTS_NUMA_BALANCING select ARCH_SUPPORTS_PAGE_TABLE_CHECK select ARCH_SUPPORTS_PER_VMA_LOCK + select ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH select ARCH_WANT_COMPAT_IPC_PARSE_VERSION if COMPAT select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT diff --git a/arch/arm64/include/asm/tlbbatch.h b/arch/arm64/include/asm/tlbbatch.h new file mode 100644 index 000000000000..fedb0b87b8db --- /dev/null +++ b/arch/arm64/include/asm/tlbbatch.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ARCH_ARM64_TLBBATCH_H +#define _ARCH_ARM64_TLBBATCH_H + +struct arch_tlbflush_unmap_batch { + /* + * For arm64, HW can do tlb shootdown, so we don't + * need to record cpumask for sending IPI + */ +}; + +#endif /* _ARCH_ARM64_TLBBATCH_H */ diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 412a3b9a3c25..3456866c6a1d 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -254,17 +254,23 @@ static inline void flush_tlb_mm(struct mm_struct *mm) dsb(ish); } -static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, - unsigned long uaddr) +static inline void __flush_tlb_page_nosync(struct mm_struct *mm, + unsigned long uaddr) { unsigned long addr; dsb(ishst); - addr = __TLBI_VADDR(uaddr, ASID(vma->vm_mm)); + addr = __TLBI_VADDR(uaddr, ASID(mm)); __tlbi(vale1is, addr); __tlbi_user(vale1is, addr); } +static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, + unsigned long uaddr) +{ + return __flush_tlb_page_nosync(vma->vm_mm, uaddr); +} + static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) { @@ -272,6 +278,38 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, dsb(ish); } +static inline bool arch_tlbbatch_should_defer(struct mm_struct *mm) +{ +#ifdef CONFIG_ARM64_WORKAROUND_REPEAT_TLBI + /* + * TLB flush deferral is not required on systems which are affected by + * ARM64_WORKAROUND_REPEAT_TLBI, as __tlbi()/__tlbi_user() implementation + * will have two consecutive TLBI instructions with a dsb(ish) in between + * defeating the purpose (i.e save overall 'dsb ish' cost). + */ + if (unlikely(cpus_have_const_cap(ARM64_WORKAROUND_REPEAT_TLBI))) + return false; +#endif + return true; +} + +static inline void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm, + unsigned long uaddr) +{ + __flush_tlb_page_nosync(mm, uaddr); +} + +static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) +{ + dsb(ish); +} + +static inline void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) +{ + dsb(ish); +} + /* * This is meant to avoid soft lock-ups on large TLB flushing ranges and not * necessarily a performance improvement.