From patchwork Wed Feb 26 12:03:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13992200 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40F9EC021B8 for ; Wed, 26 Feb 2025 12:04:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9865928002D; Wed, 26 Feb 2025 07:03:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8ACBE280024; Wed, 26 Feb 2025 07:03:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 615DC280034; Wed, 26 Feb 2025 07:03:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3361828002D for ; Wed, 26 Feb 2025 07:03:52 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id E54BCB69CF for ; Wed, 26 Feb 2025 12:03:51 +0000 (UTC) X-FDA: 83161961862.14.C53366C Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf25.hostedemail.com (Postfix) with ESMTP id 053BCA000E for ; Wed, 26 Feb 2025 12:03:49 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740571430; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=FiHnF+KuTwEjvFyPJ3PoiZHnBerVifqImn6TA6yPpT4=; b=p+MLLeTbqeTeI26XArMDjep5NhZoVO/Bbm5jh5M0RHqvegIKGXZIJf4ZYz7QHtIZsbU1Jv jDTO2F0IKP0YGaMjzxZsIezrtRko9vut3Fn5ur0az989aQw5aLJfEe2Le8HOu1eSl5njWJ 0SQod0NThGb4mkH6MmOQWjAYb/y63Dw= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740571430; a=rsa-sha256; cv=none; b=1j67Gc/GI2FZT3AFKNRNIUw8ENkX1ssKF5Lg91vv3Z9G9kVB9AEPrWvHIrXk20qHVvsfDH 7AXo1cLHpgYNO8riEECPkDZVUdxgiepYZfAzEboc2QJD+fwKNZzzfd50roUOmUTaiI/1+x Rxm6lyqoLPhCrSdzEmgsSFsf+6yL/KY= X-AuditID: a67dfc5b-3e1ff7000001d7ae-f6-67bf03226ffd From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, rjgolo@gmail.com Subject: [RFC PATCH v12 based on v6.14-rc4 01/25] x86/tlb: add APIs manipulating tlb batch's arch data Date: Wed, 26 Feb 2025 21:03:12 +0900 Message-Id: <20250226120336.29565-1-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250226113024.GA1935@system.software.com> References: <20250226113024.GA1935@system.software.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMLMWRmVeSWpSXmKPExsXC9ZZnoa4S8/50gy2LxS3mrF/DZvF5wz82 i6/rfzFbPP3Ux2JxedccNot7a/6zWpzftZbVYsfSfUwWlw4sYLI43nuAyWL+vc9sFps3TWW2 OD5lKqPF7x9z2Bz4PL639rF47Jx1l91jwaZSj80rtDw2repk89j0aRK7x7tz59g9Tsz4zeLx ft9VNo+tv+w8GqdeY/P4vEkugCeKyyYlNSezLLVI3y6BK6P3wxO2gidCFUun8DUwfufvYuTk kBAwkVh6+Ds7jP33218wm01AXeLGjZ/MILaIgJnEwdY/QHEuDmaBZUwSe080sIEkhAXSJPa2 HgazWQRUJZ5fvM0IYvMKmEocblrPBDFUXmL1hgNggzgFLCSmTdnGAmILCZhL/Pm1jwlkqITA fTaJu297GCEaJCUOrrjBMoGRdwEjwypGocy8stzEzBwTvYzKvMwKveT83E2MwKBeVvsnegfj pwvBhxgFOBiVeHgfnNmbLsSaWFZcmXuIUYKDWUmElzNzT7oQb0piZVVqUX58UWlOavEhRmkO FiVxXqNv5SlCAumJJanZqakFqUUwWSYOTqkGRtmrSaGVnJtKb742aGc825wkM8FCfIf925cL it83PGs3jztpsED/a5Fl8IbFCrMXWp5+apDw4fSEW3k6PVFT7AXzWVTMHXmsL4hJ3i+P/5Mu psF1vp5p624f66Dw3QLzBf5LfJ+4QbneTmnNJsVFh2sLTlg3hvhy2qXd2fb/4DJ2D26vAs5j SizFGYmGWsxFxYkA1yTHI2YCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrFLMWRmVeSWpSXmKPExsXC5WfdrKvEvD/dYPN0Hos569ewWXze8I/N 4uv6X8wWTz/1sVgcnnuS1eLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHG89wCTxfx7n9ks Nm+aymxxfMpURovfP+awOfB7fG/tY/HYOesuu8eCTaUem1doeWxa1cnmsenTJHaPd+fOsXuc mPGbxeP9vqtsHotffGDy2PrLzqNx6jU2j8+b5AJ4o7hsUlJzMstSi/TtErgyej88YSt4IlSx dApfA+N3/i5GTg4JAROJv9/+soPYbALqEjdu/GQGsUUEzCQOtv4BinNxMAssY5LYe6KBDSQh LJAmsbf1MJjNIqAq8fzibUYQm1fAVOJw03omiKHyEqs3HAAbxClgITFtyjYWEFtIwFziz699 TBMYuRYwMqxiFMnMK8tNzMwx1SvOzqjMy6zQS87P3cQIDNFltX8m7mD8ctn9EKMAB6MSD++D M3vThVgTy4orcw8xSnAwK4nwcmbuSRfiTUmsrEotyo8vKs1JLT7EKM3BoiTO6xWemiAkkJ5Y kpqdmlqQWgSTZeLglGpg3Hc1sXj3rrN9zTy/fJkZrDsf6uy/IM/K4+AWFHgo0m7e3IXmKk78 7devnHynG1MkufbI++cr+SVmbZ0n+dG+u/3Exq1HnfQvv99csZd3l8EMXbWIPD9Pz1C5d1ev ZMw/sy560kX9TK5bq5c2/bw/6/TmwyuL9woeevgsSPLzx8CT3ZWLFlZYSCmxFGckGmoxFxUn AgAUVfMcTQIAAA== X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 053BCA000E X-Stat-Signature: 1th9s93neie7tqkpmp1u4u3s11hsfgtr X-HE-Tag: 1740571429-258940 X-HE-Meta: U2FsdGVkX18rd4cFl0VFtStX85FYxx/NL9+yAK79jCFZLo0quwjFjXVP/N0BL1NvsVZ8pLKu2lTRMxpuVuje9iky4q/BnCyaSp3sh1pL4fqQzqk1pc8w/uaVln4asHbel3E28v68SsPHErE2gmXOh4jg8BenWr8R47jyzEJNmAj7SsOxY72j6iA3LUUeKVExK5ZZjzshpPBRDHzilY3XTeV6O1/OjnuIWrsDn5LrPud4iZJ+npxqfkkgVjqt71i31mlnboPGjqhReFZ+xhMzU9l1VCDYlp/zX5XD7hSBlcew6Pmy6Uc5vW3WBSMzQ6QjhKbzrHGrvSTNwm+nyPs2kk2Cym9fds0WEGVnOEDmOy2tjCJIgSFE4m+EPqt3R0vk5lOwDM2UOkX4JnuB2xfxNlqIAwnX5zkMXfplodeT9pzTW1PWYcVQ0uwxQmE5YxImPTDr/itA9K5xCJ+rbD5FsUVQfKbIvYQ4Eiz1YIuBlXy5wHLInFlU7e2wjWEyaadslsI5XQS9ceXmCrgQk4TiOHQe8XR4cuvM9EGd6tE8j2VAZInk50dMKLWCHpeFZXYNfoY66bmaQkEupy00ydEYTyTwagNyZ7o5r8+oFg+HVYd6cEqwM9nfNskQ0j4rHdjDWcUKWxUVmuMalPtfawDQUYzjbavzzZ5mvNP8ABxM79dY/8igtUMnlsmx3UPN2m+z05TmxzoeFkRSktuX5QNe+ufCFlaUsvuZBbb09fmzoBIo28KUiVhN98N8dv5Bh3OPCWx6J9vp18Qxj5+rfZzPExUglyIPCmdFzTgzyYThwftX0+iu5yKWsVVpyYakpltL9s0Yoi0RlOWYir7iIFbjh26fNcbjraYTyb058MH3GfwKhlI8iwJd6T6hAWNEHJGE38ro7GRMH2ugfCWLKMWSaVIqcSkRYnWr4vn0+V9+od+g/sTmpqcIKT8PHCVzY3iTd6C6yPHKeavSc9lRl3p DYkaJfQM OdjTwChxLpQHQpqfqKWXsLQIMYg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) base-commit: d082ecbc71e9e0bf49883ee4afd435a77a5101b6 diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 02fc2aa06e9e0..c27e61bd274a5 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -294,6 +295,29 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_need_fold(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm) +{ + return !cpumask_subset(mm_cpumask(mm), &batch->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access)