From patchwork Fri May 10 06:51:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13660933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04D0AC25B10 for ; Fri, 10 May 2024 06:52:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E5886B0095; Fri, 10 May 2024 02:52:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 595A76B0096; Fri, 10 May 2024 02:52:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45CE26B0098; Fri, 10 May 2024 02:52:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2898E6B0095 for ; Fri, 10 May 2024 02:52:24 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D3D9D16175A for ; Fri, 10 May 2024 06:52:23 +0000 (UTC) X-FDA: 82101567366.01.E045D99 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf02.hostedemail.com (Postfix) with ESMTP id D46278000A for ; Fri, 10 May 2024 06:52:19 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715323942; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=lYfFMbAEM1F2cBEc4miPa+Jx1WoZRiG7k2wKNFJlSqA=; b=Il8wv5/t0ZjyRwEMsThVJk/cjhlW57T3oU1J2UXug3QZPF8OXIVv4+KzORqqsdxXCzPs3e 7Qvl12U+3gTBMmJQ/Nc7+ws1zWDZUFtr2t3af1UMZOC2JuP1DgRV+BAm9If+mFUFagpdsD fcDjDlAVGU1xJH6HHwXfv3nJMTjdp5A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715323942; a=rsa-sha256; cv=none; b=nH2SYK3Hk6Vl7sdJ9eIY/+kOFiXRYP+Ul94F7O9OQsoa42b4apgTqmNok6bfFMQjX3KnYL oyq8B1F4f3pdV0yko4wVn9STM6V0TB1R30IVo0q9esuWazAcD5pQGRegY8fmAyN2olpMhJ v0IqSf5mhmAv84ZFSd9ItSvGX/BaUpI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-c4-663dc4212db8 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v10 01/12] x86/tlb: add APIs manipulating tlb batch's arch data Date: Fri, 10 May 2024 15:51:55 +0900 Message-Id: <20240510065206.76078-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240510065206.76078-1-byungchul@sk.com> References: <20240510065206.76078-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKLMWRmVeSWpSXmKPExsXC9ZZnoa7iEds0gyXrJC3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG yf4NTAVvBCrOPJ3H3sA4h6+LkZNDQsBE4serPkYYe+WfVewgNpuAusSNGz+ZQWwRATOJg61/ wOLMAneZJA70s3UxcnAIC/hL3DnAChJmEVCV6Jn3H8zmFTCV2PrvAjPESHmJ1RsOgNmcQGM+ LJvBBGILAdV0PJgItJYLqOYzm8TS9d1MEA2SEgdX3GCZwMi7gJFhFaNQZl5ZbmJmjoleRmVe ZoVecn7uJkZg4C+r/RO9g/HTheBDjAIcjEo8vDs226QJsSaWFVfmHmKU4GBWEuGtqrFOE+JN SaysSi3Kjy8qzUktPsQozcGiJM5r9K08RUggPbEkNTs1tSC1CCbLxMEp1cBYca0w+sEKkTpO w3Vc07iOhc1evMD/jf9krhcvdJ6dYjsn1/Q5SKL19iLOJd2/7vRVn9Y9MjX27Lxiw6ux28W4 3r/iWt+alCYWtsv+6nwzP/WrP5u1zI05Hq2NlgqYcWE7k9bvoD72op3qNzY/4te+lKg179Ma vwsSXWky2anGG50Cbp+7VqGnxFKckWioxVxUnAgA+SxPCngCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsXC5WfdrKt4xDbNoG8Wr8Wc9WvYLD5v+Mdm 8WJDO6PF1/W/mC2efupjsTg89ySrxeVdc9gs7q35z2pxftdaVosdS/cxWVw6sIDJ4njvASaL +fc+s1ls3jSV2eL4lKmMFr9/ABWfnDWZxUHQ43trH4vHzll32T0WbCr12LxCy2PxnpdMHptW dbJ5bPo0id3j3blz7B4nZvxm8Zh3MtDj/b6rbB6LX3xg8tj6y86jceo1No/Pm+QC+KO4bFJS czLLUov07RK4Mk72b2AqeCNQcebpPPYGxjl8XYycHBICJhIr/6xiB7HZBNQlbtz4yQxiiwiY SRxs/QMWZxa4yyRxoJ+ti5GDQ1jAX+LOAVaQMIuAqkTPvP9gNq+AqcTWfxeYIUbKS6zecADM 5gQa82HZDCYQWwiopuPBRMYJjFwLGBlWMYpk5pXlJmbmmOoVZ2dU5mVW6CXn525iBIbxsto/ E3cwfrnsfohRgINRiYd3x2abNCHWxLLiytxDjBIczEoivFU11mlCvCmJlVWpRfnxRaU5qcWH GKU5WJTEeb3CUxOEBNITS1KzU1MLUotgskwcnFINjAdD91Z9u/mx0/1X8I/jh7y7GsNDhR3v pGTVrdj47sfC1WpNtzXjCv4snDaldsakQ6vk3EJzTj7nygovndrk9n77XcHg95LLP7ce73r3 q+Lrmnsqh+YUPZ1Ut+ygosDqz/ueTDyhZBOzxK9vltMMLT2BJXWh4fHLivsPbr849+Idh61W OXbdS/iUWIozEg21mIuKEwE4VGM2XwIAAA== X-CFilter-Loop: Reflected X-Rspam-User: X-Stat-Signature: ihyfc3pocoogqhjqjaiqafb4rz7st3mk X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: D46278000A X-HE-Tag: 1715323939-752089 X-HE-Meta: U2FsdGVkX19DBOwHwzFQfwhqhcc9ceBKUtJGV49eAvjGzw2SMe46JsV7sBU/Bsel1UBnB8ATnkNJ+TfU3xzbs3wCESrOu6s7sawQ6kbYAdhd9/gqnbjKhPLF4zG5Nw8qoCGdUOXyEqioJNN1fa8aWHmQGXI6mbC02DyHoUzMDOUjdUZwYsT/Nudb7qd0/W4XeQgdgSKUYIRp/D4T4uigKpui4eVmcDfBPtfJMh1ZLv9MkaDBH1RlXI4zK10smIrxXcNT4WkRDyg5yFn3iupDPbv1a36C2a4ZeJii7pH2d2TzkmbZEJkfNzj9ZsFusR+FDKHl5wT0OCFTlG9NhE1cWc/Qr8Xu6aN9yuOAV+cq5nKm3GnVnP2YJ9o4fxzHNcCfveQ7nZXuPB8tR9xP8OMqs0QR9nOa/OhiM1xRc5kogswtVY2owd7C9Hv2rnhl0sXPrl78NNWEUkjojp9o3U4v51W5oOXsexkcnz3rtQxCvePvOo34nuNA46Xv4QUZqCODXNAewIosAgQBUzZ2fOH2CEiQhQO9OEdhxVL7vkraRlmCklXBhMT96Ryh75WFNGxIRPAwxi+aQZXAEnpAMM4hsFIC2PogNUvC9SRlsAOKXC2WMu/PMw927ptOuWz/5OPsWmIFmv+6Omvm3YBnc1DT9x1gkjGR12jj+Ra667xZc6OtiiUKIffA+ZtGhRqoq9yvb0kSvH32GE94CshTXSwCno8FxHH9HG28gYC0ZGte6oBFEnRUz7O52HCd8MjAwiylkhYx2VLP5WGAKQ9gT+nzK78K9tSOPkcaiCHe8CAE7LjuAl5x55DZMYgvWshWEn1jwWE/STV8VPZXHWvyXYoAFnNUsG6oKzLV/42Ln/KkowsWDngyTEUDVHb2rcP0IhuGsJhVgwKqk/pZ7uWfS9Tk5qnT9FxhzQJvjmDT1IDWY3mrbw1RHv+4kXMN66MPNrgLrtSPRHvEOcMm7uQVVPr U0UyPPUq 22edaWYgHfAXX7/FVWPNNEBsYkA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read-only and were unmapped, since the contents of the folios wouldn't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..a14f77c5cdde 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access)