From patchwork Mon May 20 02:17:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13667995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 193F2C25B7A for ; Mon, 20 May 2024 02:17:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A372C6B008C; Sun, 19 May 2024 22:17:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9BED96B0093; Sun, 19 May 2024 22:17:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C7AB6B0096; Sun, 19 May 2024 22:17:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4ED376B008C for ; Sun, 19 May 2024 22:17:50 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id D31641A0AE9 for ; Mon, 20 May 2024 02:17:49 +0000 (UTC) X-FDA: 82137163458.11.E602D73 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf01.hostedemail.com (Postfix) with ESMTP id 5F83240008 for ; Mon, 20 May 2024 02:17:47 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716171468; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=Ilpey0YHF3hm79X1vBIhnNe8zbnRWNG/60DUNWlxvcE=; b=ymy18wzJgG1IkbwD75kgFDT0MkmHH47y8ARYUC8x78G5mG6TCarJaQ9/7nKDeQM7v1emzO tbVl00zk4PLrQLymtJqzivkd5zustMnRmWA8og5bo6V4z0Z0uLEtPLt3IBuEfDRljZyfvp SCcGZ3B4Xba8RGjm+pF7wWq/THXvav0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716171468; a=rsa-sha256; cv=none; b=jgGEul3KpcV5vcdB41N7Q+ex7mjCKw7yMcq4qIoi/je1wznePguHpFeqy+0GFkvuH2BR7Z 2tEFj5qZX5q13TFFHAKolC4upJdNlwARDq6gqJCbydxS4PMZnoYmyNJp+z4R4bTSZhOM9S v/juA9lAZR+zd5Iea68M/M4KC3B4NYk= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none X-AuditID: a67dfc5b-d6dff70000001748-a4-664ab2c82afd From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RESEND PATCH v10 03/12] riscv, tlb: add APIs manipulating tlb batch's arch data Date: Mon, 20 May 2024 11:17:25 +0900 Message-Id: <20240520021734.21527-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240520021734.21527-1-byungchul@sk.com> References: <20240520021734.21527-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnke6JTV5pBgd22ljMWb+GzeLzhn9s Fi82tDNafF3/i9ni6ac+FovLu+awWdxb85/V4vyutawWO5buY7K4dGABk8Xx3gNMFvPvfWaz 2LxpKrPF8SlTGS1+/wAqPjlrMouDgMf31j4Wj52z7rJ7LNhU6rF5hZbH4j0vmTw2repk89j0 aRK7x7tz59g9Tsz4zeIx72Sgx/t9V9k8tv6y82iceo3N4/MmuQC+KC6blNSczLLUIn27BK6M 5evuMRZ8FKg4+m8VewPjMr4uRk4OCQETiecXfrLC2IebzrGB2GwC6hI3bvxkBrFFBMwkDrb+ YQexmQXuMkkc6AerERaIlLhyuR8sziKgKnHhwVQWEJtXwFTi7Kt3jBAz5SVWbzgANocTaM6M VTvBaoSAan4c/Qo0hwuo5j2bxIv2qywQDZISB1fcYJnAyLuAkWEVo1BmXlluYmaOiV5GZV5m hV5yfu4mRmDoL6v9E72D8dOF4EOMAhyMSjy8Ox55pgmxJpYVV+YeYpTgYFYS4d20BSjEm5JY WZValB9fVJqTWnyIUZqDRUmc1+hbeYqQQHpiSWp2ampBahFMlomDU6qB0dzQNv9bhHdLzLGn 0/LP3mnRbP8lfS/+++J/jhOnzTBYxym/38m7yez3xq+W0h4cbZ/5pjHeNr95asVZF1fGaZdc Eo/lCi9v04rnkXhfyLR5juETx8ZtB5rufFllIKhxP+8S2xWX97U/l6i+3zPpnHWrrYjX4Sc9 /3msuu7dlv3/YpfCvVvaMUosxRmJhlrMRcWJAGJSZnd5AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsXC5WfdrHtik1eawcNvhhZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+O9B5gs 5t/7zGaxedNUZovjU6YyWvz+AVR8ctZkFgdBj++tfSweO2fdZfdYsKnUY/MKLY/Fe14yeWxa 1cnmsenTJHaPd+fOsXucmPGbxWPeyUCP9/uusnksfvGByWPrLzuPxqnX2Dw+b5IL4I/isklJ zcksSy3St0vgyli+7h5jwUeBiqP/VrE3MC7j62Lk5JAQMJE43HSODcRmE1CXuHHjJzOILSJg JnGw9Q87iM0scJdJ4kA/WI2wQKTElcv9YHEWAVWJCw+msoDYvAKmEmdfvWOEmCkvsXrDAbA5 nEBzZqzaCVYjBFTz4+hXtgmMXAsYGVYximTmleUmZuaY6hVnZ1TmZVboJefnbmIEBvKy2j8T dzB+uex+iFGAg1GJh3fDbc80IdbEsuLK3EOMEhzMSiK8m7YAhXhTEiurUovy44tKc1KLDzFK c7AoifN6hacmCAmkJ5akZqemFqQWwWSZODilGhh5QlaWJ0SXfXt1V6+JVTbxymsnGeed0R3c 90163954nhTNI2XXrRy37k2vkQX/2QKnbXv9TvC0aRed+98m7Cf74tHUG1Uz3BYFParzzBUT Dlva1iExLeWCVBTrqwAp6yeBdyZubV12X9O4bv7ML5PeRysadp/WW7eql0c0TTc/8+RViwM7 PiuxFGckGmoxFxUnAgCiDQqAYAIAAA== X-CFilter-Loop: Reflected X-Rspam-User: X-Stat-Signature: gzeeou58r116jnqiu5z5e7ab4cmezq74 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5F83240008 X-HE-Tag: 1716171467-259318 X-HE-Meta: U2FsdGVkX19G/MbmnTnnyCjBZQqhptnb1iWPbVE7SkNST9x7BtXGXBBh2xCDBDNmtSy24outlGBTTx54nPXB+6hedf5FlWEv3KaXlE9I7zWaGgepgGtT1r7OdZenI0/xFbygIrslewcfgjR7V+KslSvOCxjR4R61OldkPNzsLzSiOTr2nSXkbQw5FShVmcFQvZrfGztXzUjwSesgMcK7j/XtWzHB7WV0aroL6XJrlkiu6VI4UcIYKDiPd+jznvoKqtvn9HSGCct6ArsuukI02CRzNQW4Mmmti8znr3m4xBlQlOKzjGvrd4NV1+XejaB6cqTn/vmaXmFuvk4GpDSumbugd7BpWB4rh88tnElEtKPVWmRvELQHI3zAinRpZtxp2GkABXoUVb/2eqJpv5fqGoG+Bu6MRqCfI/Ne9puGcpURvd7cEcMd7cgcWazQUyjGiTBslVeMUtf723wX0Z6rwEPVbPqSShbhcXDFWZ9zpS74Xx5rCAIhqfIkZf50i0Z28JpA00x1FrrTCoUawoey6ZIfG0e+cCzJD+dZ5cD80IefCRMrGGtzXPZmT2+ldYNDw3Qsifx+jyLptl9/jmD5UDz2Xtlg2Z/jfI/hnsEP6nqZqfqHtck7cva6muXNkERA+8l3sy6tmGdxgCU9lExNLGdUD4pt2F4JoRHL+XelDWyaHnqRhdSY7JNw2wZI5eGinl3O3i9D7+KsMs9LXJrAvpeKuJzJJPyRSX/fXLw5cG7KRx8Q7magpyK4ZW8Nxg322BDv4jFSjguGM2/CBl0GtXcvZxXBZ8PalgaynqAkXHBLhr/UipTBv1sVWz7gyIonDIqmFrfQFqAlHW6CSorYozlbwR4gKujLQH7Zhkynyf8czezZk9fQPBULFoSuapgwac8JBHKNrFonm/d7TxUFkXTaYhctBQBotiGSHnyywQUzJ/B+M7tgnK3t+BNYTVluZE09eMJZ95hzYHXLheA y6BwsNkO 6SX6A2DQZ+wVwvHtWCUOV3kkJoA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. This is a preparation for the mechanism that needs to recognize read-only tlb entries by separating tlb batch arch data into two, one is for read-only entries and the other is for writable ones, and merging those two when needed. It also optimizes tlb shootdown by skipping CPUs that have already performed tlb flush needed since. To support it, added APIs manipulating arch data for riscv. Signed-off-by: Byungchul Park --- arch/riscv/include/asm/tlbflush.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 4112cc8d1d69..480c082ccde3 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -8,6 +8,7 @@ #define _ASM_RISCV_TLBFLUSH_H #include +#include #include #include @@ -55,6 +56,26 @@ void arch_tlbbatch_add_pending(struct arch_tlbflush_unmap_batch *batch, void arch_flush_tlb_batched_pending(struct mm_struct *mm); void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); +static inline void arch_tlbbatch_clear(struct arch_tlbflush_unmap_batch *batch) +{ + cpumask_clear(&batch->cpumask); + +} + +static inline void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + cpumask_or(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} + +static inline bool arch_tlbbatch_done(struct arch_tlbflush_unmap_batch *bdst, + struct arch_tlbflush_unmap_batch *bsrc) +{ + return !cpumask_andnot(&bdst->cpumask, &bdst->cpumask, &bsrc->cpumask); + +} + #else /* CONFIG_SMP && CONFIG_MMU */ #define flush_tlb_all() local_flush_tlb_all()