From patchwork Fri May 31 09:19:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13681398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AAAEC25B75 for ; Fri, 31 May 2024 09:20:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E1076B009A; Fri, 31 May 2024 05:20:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1921E6B009B; Fri, 31 May 2024 05:20:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F26196B009C; Fri, 31 May 2024 05:20:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D7C636B009A for ; Fri, 31 May 2024 05:20:17 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9B1331A0CA4 for ; Fri, 31 May 2024 09:20:17 +0000 (UTC) X-FDA: 82178144874.03.15C20D1 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf10.hostedemail.com (Postfix) with ESMTP id 84ACFC0014 for ; Fri, 31 May 2024 09:20:14 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717147216; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=lYfFMbAEM1F2cBEc4miPa+Jx1WoZRiG7k2wKNFJlSqA=; b=lK6TI7StNPcfy7QqBgXB5MZpqG5zaMUncCP/QVtNMy8ZyhwC3VnHZRPYVQFVQ4eFXnKIUj P9oHBtc0d15R5aRIWRws56VBn3EVqrbBCmBOZDyiZ/ZFlVvpyKQTYmlsYig6Dh3ayOJVAj Qg67BPyB5COGz2JIXMlpW4JtGpS2zqc= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717147216; a=rsa-sha256; cv=none; b=npw6Hlk+2Kw0zU1nVBUkEVjocrINfJ9scMExuuu1Uc14jZSi2BkHAOhGSxk3pSjeS7NGFg +vXHrGsUAKf7fjofaobQL9TGYK5RT63xbkdXPRNh1+742+QCxCmyWM1aWh8gMlhFeWr4r2 ZWQ4drZd1DeI5DguB5/dhYDrKXgdmhs= X-AuditID: a67dfc5b-d85ff70000001748-43-6659964b11d6 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v11 01/12] x86/tlb: add APIs manipulating tlb batch's arch data Date: Fri, 31 May 2024 18:19:50 +0900 Message-Id: <20240531092001.30428-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240531092001.30428-1-byungchul@sk.com> References: <20240531092001.30428-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnka7PtMg0g+3bGC3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG yf4NTAVvBCrOPJ3H3sA4h6+LkZNDQsBEYvWz+8ww9oRPDWA2m4C6xI0bP8FsEQEziYOtf9hB bGaBu0wSB/rZuhg5OIQF/CX+7rMGMVkEVCUu76oHqeAVMJVo/vuVDWKivMTqDQfApnACTTnw 9w4jiC0EVLPofy+QzQVU855NYl3/DagGSYmDK26wTGDkXcDIsIpRKDOvLDcxM8dEL6MyL7NC Lzk/dxMjMOyX1f6J3sH46ULwIUYBDkYlHt6Aiog0IdbEsuLK3EOMEhzMSiK8v9KBQrwpiZVV qUX58UWlOanFhxilOViUxHmNvpWnCAmkJ5akZqemFqQWwWSZODilGhi1nhddOdE/89khh6lu F7/bL9I+ZbTRzuWBU9PijsntB26wnX6yTsyt7ZXu7MzS/d8jc+4x7DGRUGfa3GcQNGf5OhvD JYJrFb+LLbDRPOO69YjQhx2n7sbIHv+y4kr8F7/Qb8omyxSm9ti+MOBQ8nj++lXg7wz5pX4T yrtO3/r4brPIfo/DUtNvK7EUZyQaajEXFScCADKmY6d3AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrOs9LTLNYOZZY4s569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZZzs38BU8Eag4szTeewNjHP4uhg5OSQETCQmfGpgBrHZBNQlbtz4CWaLCJhJ HGz9ww5iMwvcZZI40M/WxcjBISzgL/F3nzWIySKgKnF5Vz1IBa+AqUTz369sEBPlJVZvOAA2 hRNoyoG/dxhBbCGgmkX/exknMHItYGRYxSiSmVeWm5iZY6pXnJ1RmZdZoZecn7uJERjEy2r/ TNzB+OWy+yFGAQ5GJR7egIqINCHWxLLiytxDjBIczEoivL/SgUK8KYmVValF+fFFpTmpxYcY pTlYlMR5vcJTE4QE0hNLUrNTUwtSi2CyTBycUg2MmxuYsyT7ap1nPP+7V7Xyzpl9Sf/tTm84 v3r/p6Kfc2bMvhWe+8R51ctnF+M3eDnkqbouWPFyqmZS+7/fmfUqdbsWbU1U+Jeiq8vKXKAw IzBshs0+p38RIZMm3J8mWm7eGfi2Q2W6lFnO9LBQbdaoiae/6EhrbmlXP74y6UlsyCSBlUWX yo8dUWIpzkg01GIuKk4EALuE7ydeAgAA X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 84ACFC0014 X-Stat-Signature: 6kdmxtk97gk3r4ixq6nd7ang6q1qz6fj X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1717147214-885558 X-HE-Meta: U2FsdGVkX188gkG7/MI6X8ft3Wy9PNQ62p295mTi6kg3DFc5IICQ6qATD8WiODBht83SSMX7vRZRpFD1TIuePFu8LymTkgwCw0CUjHiaQZKYIcxFLEDiHnUVYpHf/4VC41UC49n5S7d4gf/zAy2hKizAgWnM8Ne+bhNdD0YZ5RzeSJLYJNtJkB3R2eKrBSE1/dM03lxYt8lAbUpA+FoLGYJOtYB5jaur08xwJsKsS5P9Wnx7F6Q3HM3WUcrf7t7VzzuJqppwWTHE0C9hqS47AWnMhFAZoGvI0EaQ8LuKPlgwPhoUlbjYqX7hh2YqqTKQyb+R+rc8NM9nL/8zmnaV0aQ4pQEOgp7atBhrHAE4AEXm+QKz5SaHeuqeysjp+UKcUZ8KQ+546bGvjwdqs3UxVEu9S9RUb4sn8BuyknHzK5KVurZK12o3CXdGEhEKFgEXQ111PsqgxBeQWgj1XJFRN/rzbfVIkPjSvIcsu2dGVFnpAp4wJ1QIkOQYiFqGSAw4Oxl55uNMdvHfo70YxS+OIvr01qVPnAY+NpyjLEe8IlgWkay9NDy1ST1Iwk85mFg8L7qAcoXwmfxW/6dSyQA27movLDiPt28j6jM59bbjIdEs99UU4gbUi/JU0OUBuWN5F+zF8gX7wP7EzSTj7DcEHI+8RrCZSzzkc0yFXNJEGXCpYCKM5mhU84VhYfh2Ts2VyFlgcuxbkTKIvt+d+49FowjYn8bYEwEOfA1YB0a59cjQ1e/uA17zQha0cQ1ai5f19cldtFvT8FUBVvkY32kdy3tnOasyuCfFU7XWLh52SySuANu1098z+W4Le7ddcMCq1ow3FPRT8ZeAmsxM93dmTPz+plsZuapjflPXe06D390p2/LVLpjQ9cq8e2Vh97h0EHS1aO8slrqOYiou9IPHED8dw5G/KfQjNqBMDr1vCl+FMCuEKTOUbwqCfPMsg29KIoMRDu4EdQJij1AviYV rFBTBq85 2vzeS1yoc/G2pruWHAbVAwm15Yw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..a14f77c5cdde 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access)