From patchwork Fri May 31 09:19:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13681397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10B1CC27C44 for ; Fri, 31 May 2024 09:20:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 95BA66B0099; Fri, 31 May 2024 05:20:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 90BB76B009A; Fri, 31 May 2024 05:20:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D3806B009B; Fri, 31 May 2024 05:20:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 607686B0099 for ; Fri, 31 May 2024 05:20:17 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0C7F8A10AD for ; Fri, 31 May 2024 09:20:17 +0000 (UTC) X-FDA: 82178144874.06.B25818A Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf22.hostedemail.com (Postfix) with ESMTP id 97BF8C001A for ; Fri, 31 May 2024 09:20:14 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf22.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717147215; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=2xyy9kkcj1Tl96/Je7+DpJ1ey8tWADMcAHlkm2MEIw0=; b=jCYnaoIxBoU8EaLwTgB0MMX6FHE8a5GsiL/jnDNy2hrbLYk+SLTZKgLLFBd1RL7Jr9oxF0 thWAizQ2BSn8Rp7XGxYe5KWhc6v1OQSPDIVmHId1SpcB8LRLAtrFZM3C7Q6EpLKjRhnCKK qsM9VyDlo5srCVbaNq/BF2aMWEIQDk8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf22.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717147215; a=rsa-sha256; cv=none; b=xLJfWSG2R9pJ2DGwcZYpWBYTyd6WZ4u0OfuLZAhe7DXlaFollammFx94EF2EvD8+AIhN1u HXH/yE7iqECLMFj2e82J/3eRlGWKiuBVD2ppVC7lFSpVGOBQ0x9tHJDRT/6ojY6bkSCxnw Kn/kXQznmvWHDo4klVn6fbWFkHhc1UM= X-AuditID: a67dfc5b-d85ff70000001748-4d-6659964bdba7 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v11 03/12] riscv, tlb: add APIs manipulating tlb batch's arch data Date: Fri, 31 May 2024 18:19:52 +0900 Message-Id: <20240531092001.30428-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240531092001.30428-1-byungchul@sk.com> References: <20240531092001.30428-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrKLMWRmVeSWpSXmKPExsXC9ZZnka7PtMg0g/fv2CzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG xn1vmAsmC1Zsff+BsYHxIF8XIyeHhICJxMXG1+ww9rypd8FsNgF1iRs3fjKD2CICZhIHW/+A xZkF7jJJHOhn62Lk4BAWCJJ4u04BJMwioCqxbuNyJhCbV8BUYuKZ46wQI+UlVm84ADaGE2jM gb93GEFsIaCaRf97gWwuoJrPbBLbV35ig2iQlDi44gbLBEbeBYwMqxiFMvPKchMzc0z0Mirz Miv0kvNzNzECA39Z7Z/oHYyfLgQfYhTgYFTi4Q2oiEgTYk0sK67MPcQowcGsJML7Kx0oxJuS WFmVWpQfX1Sak1p8iFGag0VJnNfoW3mKkEB6YklqdmpqQWoRTJaJg1OqgVHSue20+pTPDu9a f0uZBXKyFt8yuHklfinDHp3F6X12OhO7lP2bNkxqiPwY+E0wopuVs3RBXm7BxX2tV3Yu9eE+ lWD46FbQjl0iQr9tQpUYHd2dUh51VDfM+W9reqbK8OF3nt+PTgin1MbIJHfdNn8jsMSY68Vr ixV81xfrLJkyMSxCuV3qnhJLcUaioRZzUXEiAOq8YAd4AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrDLMWRmVeSWpSXmKPExsXC5WfdrOs9LTLNYNIhS4s569ewWXze8I/N 4sWGdkaLr+t/MVs8/dTHYnF47klWi8u75rBZ3Fvzn9Xi/K61rBY7lu5jsrh0YAGTxfHeA0wW 8+99ZrPYvGkqs8XxKVMZLX7/ACo+OWsyi4Ogx/fWPhaPnbPusnss2FTqsXmFlsfiPS+ZPDat 6mTz2PRpErvHu3Pn2D1OzPjN4jHvZKDH+31X2TwWv/jA5LH1l51H49RrbB6fN8kF8Edx2aSk 5mSWpRbp2yVwZWzc94a5YLJgxdb3HxgbGA/ydTFyckgImEjMm3qXHcRmE1CXuHHjJzOILSJg JnGw9Q9YnFngLpPEgX62LkYODmGBIIm36xRAwiwCqhLrNi5nArF5BUwlJp45zgoxUl5i9YYD YGM4gcYc+HuHEcQWAqpZ9L+XcQIj1wJGhlWMIpl5ZbmJmTmmesXZGZV5mRV6yfm5mxiBYbys 9s/EHYxfLrsfYhTgYFTi4Q2oiEgTYk0sK67MPcQowcGsJML7Kx0oxJuSWFmVWpQfX1Sak1p8 iFGag0VJnNcrPDVBSCA9sSQ1OzW1ILUIJsvEwSnVwHg42PNpzP13QlUnL8yfe4BVVKN3scNv tu0GnCuSvDTrjxv9kf59MuV2NMsyI6NdByeV33m+4WrHMf/5vDfeSTO+D7SYdfWx1+3XaufN dym+lU/8lmS7j/nU0nCvKpsOmw3ezQwiQvMXH62a1BxZrPZH7ejZpFlrl876UldrnHv70S73 RfrcLuVKLMUZiYZazEXFiQCTcEXWXwIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 97BF8C001A X-Stat-Signature: k6hbebuji6nzwqu7trm81ufekmqae58s X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1717147214-230739 X-HE-Meta: U2FsdGVkX1+L3ckkmWdGEToTCGS2vZTkoiGTSN67e0f1IZZSZRP/0E1m+Az7GhAOC9PqHQd3qbiCCL03Se5zeqT06xo6x7RGTuP4ukK+7Azn4/Kvue3nY62YumhK+724aXD/lYQOARjc6lK8v/CXv0qAvA9ov5zpOJv07fzPDiCsiiiP69nNApX17bCEs2sKVwXUYsBHdWgyRU3qbHC3c54MiwdMcRLgqnOoz5nn1XsfrHAt7dDJLMD85GeXxLp/lNNlOalZX6Mso8uZUaWa0UEUFFqbMZdmcL6L966spn/G7wCZVGXszVHetHGd2YEA6nuJ9vDShRNiy0K7hYc0Hzk3sWD5rRwicqYG8klYydxokqc2v3XIAaCiq/3JW1DhnuOFMtv9LSrSbt4Dtu5Zq+pCFEEH5t5KJS1VKzxFsSWi/pekKJMT/+RR6VqkeskqQqdjMaSWpJSHA7mgQnpCCPxRy17dc09Nu4zNqEB81qplnOF3UjPtgchAupb0c/pIoB5tXHdGCaljwXmNV01NG1AxT72y1xD0/GUyR3Pj2sD40iet/q5uahRwIhLDCEeGLEK95FuErzAzuUGV9kEmnRNz3UOfOAwrVuvTVBaRLkMX49oiyVhXdkGF42CR1FfJ8MVleb9k4O5WXSCGF88Q7g2/1CXnjC0QBJFs1V1dojqMS5Iq37Hq0eEmpYFpU5c+PkmyjIQy+jFlJJzHbB6lxeKMnNcNn7IQqgNERtVEcwympt9WtCj/JYVgWBs+7h90GDZteQga3WkRscC6hSau7Lo2Rpo6N8/KTXR93AJGJWJzhnToFcyXlQZ0e1Ak/9oSQXFkwfNEBTOCdExX0D3eqIlhhO66G9AF4WfQ/6Xf8mPOGFoLkZbqVNae4U7aM4FxLQkORKSju/uWYb4AnEggCfbG6y0NWUjoLf+7AhzA3tDkQ6Y9MQJQy/QN8SHLZSwTUeQb2ogRNy92HQ5yr1d s3pVPWbj tjuJxcigqV6gcGvkuhH0/1YfJww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for riscv. Signed-off-by: Byungchul Park --- arch/riscv/include/asm/tlbflush.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 72e559934952..c0bfb9b2bf54 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_TLBFLUSH_H #include +#include #include #include @@ -65,6 +66,26 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); + +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} + extern unsigned long tlb_flush_all_threshold; #else /* CONFIG_MMU */ #define local_flush_tlb_all() do { } while (0)