From patchwork Mon Jan 15 08:19:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13519379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EDDBC3DA79 for ; Mon, 15 Jan 2024 08:20:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BF67D6B008C; Mon, 15 Jan 2024 03:20:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BA82E6B0095; Mon, 15 Jan 2024 03:20:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A47FD6B0096; Mon, 15 Jan 2024 03:20:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 92B756B008C for ; Mon, 15 Jan 2024 03:20:08 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 61D36407F8 for ; Mon, 15 Jan 2024 08:20:08 +0000 (UTC) X-FDA: 81680847696.15.122806E Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf10.hostedemail.com (Postfix) with ESMTP id D50CDC0018 for ; Mon, 15 Jan 2024 08:20:05 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705306806; a=rsa-sha256; cv=none; b=wqlUpJVzd3nL5/hAs0cjkVqxN0nbQ3Nj+ROXuXjlcvkyblA2BktP7PblQP8WbIpXmip4aQ 1LC/axXkHtYYaj6WBU/YKXmjlRockzFF+69X79Zl8Z9liaXnmTah6S6V78eDYdPx6jO+wT b0z7lJdRIuHnTKvSqq2XUECFIPxKUw0= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705306806; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=eRD1/8o+qtEOTAKMY7ucioDJJtMI3+WpTL+ASvHX4To=; b=1PizzkrqU4xFY8AIWsdguQnyJGWltW4M8qol6Vvd9sw/Jtq5wcSmtz0fL+rIZtc3b7ZWog 5Ogtb/vFUrru3TogQLxImRUUZ8E8/sEhRJiG4LacPeK4VeGDSXgQ8Mdu1es6UNTKNCWzyX KE7Zre8RDaIMrdTAX6XG9RyAHFvJU2M= X-AuditID: a67dfc5b-d85ff70000001748-75-65a4eab4da69 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v6 1/7] x86/tlb: Add APIs manipulating tlb batch's arch data Date: Mon, 15 Jan 2024 17:19:47 +0900 Message-Id: <20240115081953.2521-2-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240115081953.2521-1-byungchul@sk.com> References: <20240115081953.2521-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRmVeSWpSXmKPExsXC9ZZnoe6WV0tSDXZN5LGYs34Nm8XnDf/Y LF5saGe0+Lr+F7PF0099LBaXd81hs7i35j+rxflda1ktdizdx2Rx6cACJovrux4yWhzvPcBk Mf/eZzaLzZumMlscnzKV0eL3D6COk7MmszgIenxv7WPx2DnrLrvHgk2lHptXaHks3vOSyWPT qk42j02fJrF7vDt3jt3jxIzfLB7zTgZ6vN93lc1j6y87j8ap19g8Pm+S83g3/y1bAH8Ul01K ak5mWWqRvl0CV8bt2y+ZClbxVSz50c7WwDiNp4uRg0NCwERiU4NxFyMnmLns9nJGEJtNQF3i xo2fzCC2iICZxMHWP+xdjFwczAIfmSRWf+9gAUkIC/hIvG16CVbEIqAq8fvGSTYQmxdo0Jxr DSwQQ+UlVm84AFbDKWAq8e7jcXYQWwiopuXuOlaQoRIC7ewST1adYodokJQ4uOIGywRG3gWM DKsYhTLzynITM3NM9DIq8zIr9JLzczcxAuNhWe2f6B2Mny4EH2IU4GBU4uH98XdxqhBrYllx Ze4hRgkOZiUR3uo7S1KFeFMSK6tSi/Lji0pzUosPMUpzsCiJ8xp9K08REkhPLEnNTk0tSC2C yTJxcEo1MHKmheSanGLfknzfh89jQ/eZsjpmuQmx9eud/hVnJZtsZ/kQe2OFs8FZ9mXLO9t5 1ut6+O5dG1KaLrp01cxoG0b7I23TWraZePttCe6JLrdg+WR3j9lw77Q5v7fFqU/82enPlnfM aG1vteG3mpMHBY5V8nhJ5Qp9dXwoLfizxfNee7/Du5PnlViKMxINtZiLihMBByu8QYMCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrBLMWRmVeSWpSXmKPExsXC5WfdrLvl1ZJUg9WvGC3mrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XhuSdZLS7vmsNmcW/Nf1aL87vWslrsWLqPyeLSgQVMFtd3PWS0 ON57gMli/r3PbBabN01ltjg+ZSqjxe8fQB0nZ01mcRDy+N7ax+Kxc9Zddo8Fm0o9Nq/Q8li8 5yWTx6ZVnWwemz5NYvd4d+4cu8eJGb9ZPOadDPR4v+8qm8fiFx+YPLb+svNonHqNzePzJjmP d/PfsgUIRHHZpKTmZJalFunbJXBl3L79kqlgFV/Fkh/tbA2M03i6GDk5JARMJJbdXs4IYrMJ qEvcuPGTGcQWETCTONj6h72LkYuDWeAjk8Tq7x0sIAlhAR+Jt00vwYpYBFQlft84yQZi8wIN mnOtgQViqLzE6g0HwGo4BUwl3n08zg5iCwHVtNxdxzqBkWsBI8MqRpHMvLLcxMwcU73i7IzK vMwKveT83E2MwPBeVvtn4g7GL5fdDzEKcDAq8fD++Ls4VYg1say4MvcQowQHs5IIb/WdJalC vCmJlVWpRfnxRaU5qcWHGKU5WJTEeb3CUxOEBNITS1KzU1MLUotgskwcnFINjLtmvf6U1GIv 4rvzC98p3/BJJ19arL37uFbRSl/dp/zHu6SePwxO4t9nhM2yOzQtaFtS8LvHJpqbfPhV7/jN 4vq65n7Nhgdfoict2ypnmriMt+8y9+NrTLv6N34rW26r5bxvadM6F/l1K/N3b+C5+Pr6GsmT jyZ/Y51lsswy8s37qunmGq51bvFKLMUZiYZazEXFiQBY1mM4awIAAA== X-CFilter-Loop: Reflected X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: D50CDC0018 X-Stat-Signature: yn3cbogw51o3fjke99d7tzj4jon5prmd X-Rspam-User: X-HE-Tag: 1705306805-900750 X-HE-Meta: U2FsdGVkX1+sTLR9TrOHYnY/zjhATIw48FyOGrXt7KR3Wv1J5OiROiO4i92NY1VMqJWEocrpuN4IJ+vNL6vM4LJS0AZuOTNMyv/aNSh9WAn0+WBz9aK7tnkXnuwTjAgJogH9c3RioOTXLpe2qwK6weYHpF+QQ1HHVtv7TgXLxOOLzfVwFB0Xc/J/kgPUNwnS9PKua33dts0z+IY4xQVcJPc3l3Pkj0NV2v1MNDg4zcJW6vpiKQz+Dn7SF88GKmYmf0FmaUfBoec4pv8sRtkTDfb4visj/LQCjAeFW5uvhp2rCG05y4xGua3lALv6M58Hpy4yYNQnpf2GZYZLvF35WqAtVvbPdLA/yPoBExMrgrBipiOU+KmbT661Fx7RjtH9Tolyijq2uN6dmNsBUGLA+soctbhBlEiS27H3A2p+CRbDhGO53lSSRyCmCKIO5qRVLWPYvj0Mg+CQTN7dZMhtddH0fKgw8U5rpZjJaXJ2pTGA0L01G+0sOaqn77Iom1sbXn//c7JwDnCxK6Ht/rNqmTsb/LqHSMWKaEWgq/Wn0THOMk0+7ofMZZmAWs04Jm+c/4ZxNno5Sza4gwY5PicAfMlQH+FMo9MukzsnhcZEnf8XV+70OKfon92Zx/sNnNujpvrkMnbr58vYyCWlyG3sxDxlaOrTFNelnyM9NZPYDQ60DOS4OFcu626T43JcJ7VRrRL6feOb8iQj3QN79dQxugewi+y76FLu/Ev8+Jpz/i0p0V2Z4t15KtZlzrN9F/QYxu102BwFedx3DlpJEluXuxTyYd6fXgVtPZ2RXlsUSS+Divay7MvmGod+15gZrr37+53zjWeD4cPNgxEHnJ1LNq2WcyqLbKU+ODpfY+EHMhSNEODa58ItMbKYYIGBZdyt32xfISGKQhUnfspWuhZp66ctUmeMtWSwsQdwmtfiSZUPF1HodgVIhhLiYxnitnmvPKEYjHjfm5mikV9yFa7 hr4af4Cr IJw/vx3UVpQWs6L3BVa0/MpLHaA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a preparation for migrc mechanism that needs to recognize read-only TLB entries during batched TLB flush by separating tlb batch's arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. Migrc also needs to optimize CPUs to flush by clearing ones that have already performed TLB flush needed. To support it, added APIs manipulating arch data for x86. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4..fa7e16dbeb44 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -293,6 +294,23 @@ static inline void arch_flush_tlb_batched_pending(struct mm_struct *mm) extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); +} + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access)