From patchwork Wed Feb 26 12:01:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13992170 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BEB4C021B8 for ; Wed, 26 Feb 2025 12:01:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D177E28001A; Wed, 26 Feb 2025 07:01:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CC82E280015; Wed, 26 Feb 2025 07:01:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B682028001A; Wed, 26 Feb 2025 07:01:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 993E2280015 for ; Wed, 26 Feb 2025 07:01:47 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 51C381413C8 for ; Wed, 26 Feb 2025 12:01:47 +0000 (UTC) X-FDA: 83161956654.29.B72B8EC Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf02.hostedemail.com (Postfix) with ESMTP id 0BDE580004 for ; Wed, 26 Feb 2025 12:01:43 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740571305; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=esqmxSvdXZ0/fOiVv7mAv4zTnaRTKUcehwals+5uS4s=; b=Xmd5s4dFxZm3y5YNOrGg7t8owdpJ72XxXFk2rcdF4wUcdX8K8MGCDRYp8CU0TccrJoAA4Y neeIXYxBpKJZeurzUJ2JGmlrZmWbptX9Qb9wcC4HUwFCl9tRLqEP8hL71m/u/085ltdsOD pAjv341FF/+mZq7HLyFRvETppKqIYMA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740571305; a=rsa-sha256; cv=none; b=ksASvRMcTAgfu9dm0Qots5/8GztXXr5hb3cdfw1R4xkbPlk0y3HjNQ650HERqDm7F+TS/T WkwwRPTECxSQ7yRJHDVrrOUAnvwyB7R72dG36DrwUeh9NpTSIyZv1NBqAq5noig0257kf4 84mqA1+nDGba6SAqwxSTZ5oh61QK8XY= X-AuditID: a67dfc5b-3e1ff7000001d7ae-c7-67bf02a62d46 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, rjgolo@gmail.com Subject: [RFC PATCH v12 based on mm-unstable as of Feb 21, 2025 01/25] x86/tlb: add APIs manipulating tlb batch's arch data Date: Wed, 26 Feb 2025 21:01:08 +0900 Message-Id: <20250226120132.28469-1-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250226113342.GB1935@system.software.com> References: <20250226113342.GB1935@system.software.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMLMWRmVeSWpSXmKPExsXC9ZZnke4ypv3pBjdeC1nMWb+GzeLzhn9s Fl/X/2K2ePqpj8Xi8q45bBb31vxntTi/ay2rxY6l+5gsLh1YwGRxvPcAk8X8e5/ZLDZvmsps cXzKVEaL3z/msDnweXxv7WPx2DnrLrvHgk2lHptXaHlsWtXJ5rHp0yR2j3fnzrF7nJjxm8Xj /b6rbB5bf9l5NE69xubxeZNcAE8Ul01Kak5mWWqRvl0CV8a1PbMYC54IVdzfVtjA+J2/i5GT Q0LAROLSlGvMMPbGi71gNpuAusSNGz/BbBEBM4mDrX/Yuxi5OJgFljFJ7D3RwAaSEBaokbg5 /R8TiM0ioCrx6vRldhCbV8BU4vGXfnaIofISqzccABvEKWAhMWveAbB6IQFziTOP/jKCDJUQ uM8m8XXDWkaIBkmJgytusExg5F3AyLCKUSgzryw3MTPHRC+jMi+zQi85P3cTIzCol9X+id7B +OlC8CFGAQ5GJR7eB2f2pguxJpYVV+YeYpTgYFYS4eXM3JMuxJuSWFmVWpQfX1Sak1p8iFGa g0VJnNfoW3mKkEB6YklqdmpqQWoRTJaJg1OqgZHloKphXtYzronKui9LV8Xmy+qlaD5akjqj Y8KdTuXTXqfvWWoUu8xoCVJKmcD18ltdltlneSmGi+6hL+9+jojRjXNYoKhhGrd5CvdBxVrZ c9/nOk8MslP5NXnBAsZV97/ZZW7YJ2ur8dzu5zW+6TZHV/JbslRe+rkyxi6HS+VPxdzFHdMs 3ymxFGckGmoxFxUnAgCaPPDpZgIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrFLMWRmVeSWpSXmKPExsXC5WfdrLuMaX+6wadX7BZz1q9hs/i84R+b xdf1v5gtnn7qY7E4PPckq8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/f8xhc+D3+N7ax+Kxc9Zddo8Fm0o9Nq/Q8ti0qpPNY9OnSewe786dY/c4 MeM3i8f7fVfZPBa/+MDksfWXnUfj1GtsHp83yQXwRnHZpKTmZJalFunbJXBlXNszi7HgiVDF /W2FDYzf+bsYOTkkBEwkNl7sZQax2QTUJW7c+AlmiwiYSRxs/cPexcjFwSywjEli74kGNpCE sECNxM3p/5hAbBYBVYlXpy+zg9i8AqYSj7/0s0MMlZdYveEA2CBOAQuJWfMOgNULCZhLnHn0 l3ECI9cCRoZVjCKZeWW5iZk5pnrF2RmVeZkVesn5uZsYgSG6rPbPxB2MXy67H2IU4GBU4uF9 cGZvuhBrYllxZe4hRgkOZiURXs7MPelCvCmJlVWpRfnxRaU5qcWHGKU5WJTEeb3CUxOEBNIT S1KzU1MLUotgskwcnFINjHcYLU5lOnoeZC5jv7Q4fK11+SGxK983+Jud65i4O0pt7bXcyZdz GSY8TtfgFLEo7GblPVvxJDiUofrqjX7XCQIzr+ZpNZXd8DIJ2bvPf8apc7eeTok+LrNSuOD7 zBNrU9b01fMnzpZdkfLk47tVS9/v6dmil3fcsdl5wcuJEeEZPOfdUwreFSqxFGckGmoxFxUn AgCLgQj5TQIAAA== X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 0BDE580004 X-Stat-Signature: 1ez98hjj3kkpgghfrnzbxso1h4a54pc8 X-HE-Tag: 1740571303-992041 X-HE-Meta: U2FsdGVkX19PBigkbyvb/UzZlHKUD8cs1WGdmNn5na2xf5H7mlVVT1jMlRYBdiK8VApK/azAcr5ms30sOZQRvR0vm24G/gVZKtyv95v5A2N7fBAkqBgdvoo8dFLpvJYp9Y+Ge/JV9rRYvnb6HiErvrxVsYgHRwwWuwAVkAYu7WUl7Mweqfk9LF16Ki4PbkN8BLcsapa93IzN/x8sDr2WD/JBaahK5KsMYseqzqcCBSj7MsB/mAZbg9D9EtIPrJgg78XyNx2u5WqXd73Fu9BahMErtEwxGuH65oexRM3wFedV5HfqAh+5gguqFeNAxlGcBbwKT07cEc3YHokJN8GJKoFPoaLCz9WFb6QzRUgjiIShEF/ELO6IlHCVJkRChwJM1rjz/P+djhAWSOUTXN6l3MlOt+IOmVbrTHt714/7r86VdJ2cSNE2oqarbhQhmwGEJ9U7x2wciEYwANcqq9Ylt1svk1riALtbqOE7c8RJIJQlIz2xSY6cmEen5pWDC11SxFsK6wbKRVCQP4IhTglJogPRX2zWvI1wg44hTFm3oZqnwlNbbHYWYlxvkrhlJAos2FjbsuGcZP4FpgImc8y/N1Mww+iFUy4YV0g3ePQOM8BcbyP0Gk+NlnJJyR2oUUD5rus+OfHVoAnUtAjbAWm0LbWtYFJxj39Ish04uDr1lUbzTStFpr6sq+U6Qvs713HgBtalkVHBshjnpy4vZ2zwvadP2JW07S5IGBBpcKqCEdL1LKNXZV0TMynSk9nsIA44lKyTPxh2k4Zsj8PZEgMlHIFHupptJs0ZWcvMKYxJhUKCtOrmZ30iLHXU4nOMYvJjpXAdBQasU7VZ5fR1H/eBsYTSbmRT1xfbru+Tp96GxiU8jahpMN+iN4ydFMF1ZLwmHV7ph7X4lgcsZvWNHkIECtpO7pKTLmplb8J+UJoQLqv8SU2lI0OEULZaxGqM4EQeJjDclciLqH9GKyUnZoF Ek3vZXih 4CU9oM2W0SC9ZxAQ2B5FTTCYxsQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) base-commit: f7ed46277aaa8f848f18959ff68469f5186ba87c diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 29373da7b00a6..52c54ca68ca9e 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,29 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_need_fold(struct arch_tlbflush_unmap_batch *batch, + struct mm_struct *mm) +{ + return !cpumask_subset(mm_cpumask(mm), &batch->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access)