From patchwork Thu Aug 17 08:05:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13356107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2DBAC2FC14 for ; Thu, 17 Aug 2023 08:09:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E90D528003C; Thu, 17 Aug 2023 04:09:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E1A1E28003B; Thu, 17 Aug 2023 04:09:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6B8B28003C; Thu, 17 Aug 2023 04:09:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B410B280039 for ; Thu, 17 Aug 2023 04:09:02 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 857EF809C3 for ; Thu, 17 Aug 2023 08:09:02 +0000 (UTC) X-FDA: 81132870924.04.C165919 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf25.hostedemail.com (Postfix) with ESMTP id 84AC8A001F for ; Thu, 17 Aug 2023 08:09:00 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1692259740; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=CI7BtFpMpIaaOCGYgXUvB3zymfhIaAUcP12OXiwAjus=; b=EOGr9wF0XOLv4sHykC+36oFZMvs6g18h33BEMG/hNF0TRaWu5tlZ2vDFf8xmeiMFobLv03 j1dIMarODuGTCEPOQ1GXidfWyyq8ATUYiXXNNfma5foVnktyfyGE2WaizFpp9Pvvw6wb7D +zj7fBC3GP+ETW3FMIYcDupUtDTA7rE= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1692259740; a=rsa-sha256; cv=none; b=2wrCp0Yh9uP6PpXNt6++D0ntKmeErRbFCiX/ZsyMH2aRAWYtkcVjZsm6IRTB6zaOVkkjFX MgH3cF/yb+zRKcGRCb74A9zyegUFPGXPwymCcKkfs6X1sJ2bkT4NuPAJMpascE4ALm9kVt KzYLrzKEIITSvOJJH72MhDh0OOrjins= X-AuditID: a67dfc5b-d85ff70000001748-cd-64ddd598b1b7 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, namit@vmware.com, xhao@linux.alibaba.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, dave.hansen@linux.intel.com Subject: [RFC v2 3/6] mm, migrc: Skip TLB flushes at the CPUs that already have been done Date: Thu, 17 Aug 2023 17:05:56 +0900 Message-Id: <20230817080559.43200-4-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20230817080559.43200-1-byungchul@sk.com> References: <20230817080559.43200-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrFLMWRmVeSWpSXmKPExsXC9ZZnoe6Mq3dTDE7M1bWYs34Nm8WLDe2M Fl/X/2K2ePqpj8Xi8q45bBb31vxntTi/ay2rxY6l+5gsru96yGhxvPcAk8XvH0DZOVOsLE7O msziwOuxYFOpx+YVWh6L97xk8ti0qpPNY9OnSeweJ2b8ZvHY+dDSY97JQI/3+66yeWz9Zefx eZOcx7v5b9kCeKK4bFJSczLLUov07RK4MpYdkS64p1Axue8EYwPjHakuRk4OCQETic0NU5hg 7MPLLrKC2GwC6hI3bvxkBrFFBMwkDrb+Ye9i5OJgFljGJHH3wDmwImGBSIl3K9rAmlkEVCUO 31kK1sArYCoxe8lPNoih8hKrNxwAinNwcAIN2vxXFSQsBFTyYe86VpCZEgJn2CSmbZ7BClEv KXFwxQ2WCYy8CxgZVjEKZeaV5SZm5pjoZVTmZVboJefnbmIEhvCy2j/ROxg/XQg+xCjAwajE w+uw606KEGtiWXFl7iFGCQ5mJRHeHt5bKUK8KYmVValF+fFFpTmpxYcYpTlYlMR5jb6VpwgJ pCeWpGanphakFsFkmTg4pRoYXf75bBe89vrhtJhWGcu90yxfHKrkkzZ7sn72XYcjTpwrPT3e f3yheun5kbblwobtpdvC3zXwdMntiJx4R/adxYXLrB8aGv0Zo473KXxO8xZexrS/yvar891T TQVxvfN5e4rkct+uiN1y2k68ctFCm76HFQKbtmfe36R7Q6JZkFlsRX9ewu3dSizFGYmGWsxF xYkABxT0Ul0CAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrILMWRmVeSWpSXmKPExsXC5WfdrDvj6t0Ugwd3lS3mrF/DZvFiQzuj xdf1v5gtnn7qY7E4PPckq8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlc3/WQ0eJ47wEmi98/gLJz plhZnJw1mcWBz2PBplKPzSu0PBbvecnksWlVJ5vHpk+T2D1OzPjN4rHzoaXHvJOBHu/3XWXz WPziA5PH1l92Hp83yXm8m/+WLYA3issmJTUnsyy1SN8ugStj2RHpgnsKFZP7TjA2MN6R6mLk 5JAQMJE4vOwiK4jNJqAucePGT2YQW0TATOJg6x/2LkYuDmaBZUwSdw+cAysSFoiUeLeijQnE ZhFQlTh8ZylYA6+AqcTsJT/ZIIbKS6zecAAozsHBCTRo819VkLAQUMmHvetYJzByLWBkWMUo kplXlpuYmWOqV5ydUZmXWaGXnJ+7iREYkMtq/0zcwfjlsvshRgEORiUeXoddd1KEWBPLiitz DzFKcDArifD28N5KEeJNSaysSi3Kjy8qzUktPsQozcGiJM7rFZ6aICSQnliSmp2aWpBaBJNl 4uCUamCcmfprh7SOzUXlmHMuxVOuNJnK5DTzyDXMUJVT5W3P/P1wRvSHrZ6rr9y8GC25bu60 PtvjD96/c+Vi7Tx7+5eBgsstd0OGK+9r4yMuGHKd69RmfWRqLhWm8D7nYrWe5gGVD4oz6g7v 6Fix4WZ53AbnqrvWHgHF27P+T75mu+uiu1PIpotT1yxUYinOSDTUYi4qTgQATWrKskQCAAA= X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 84AC8A001F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: q6ypfahzdwbyk83biw1469od8oktojfx X-HE-Tag: 1692259740-366133 X-HE-Meta: U2FsdGVkX18MTjEuLSQBWLhyT+1yp246NZ+ucmC6Hw2G2Oh7KU4pZoOltKHXIzAzypLI20BoSQ6wfJm85PtbBZKq4WNsOJxYeMdBASQJT0YT8g3hnN9nAcPi8L59USCD8MdQNtaWjB60YoJc59RV+VeMT/c8PVRyQzsfWUSf3UKCGtyzCBxjszEcxA715ycVFBxGPAI4c2YbgG8tTXBwfcMn4LA6zdwDoG2bKk1QOTfOGWv/go1BCkWGHgZjdl3J/caxt5mj62GbxgzTc/MofXJ/DVWikjXim0cqBDdyFjRB6PiYGGOVK9TqtugnR3UZDrkIU26IMtrJqDLP6cM0s1+f8wMGw6VuYGesqszszCJNOgtyvroaXvdm3OTCNXl77JYIE2OfQP/hvlnqn4avGs8mXZFc3ilGTvoAS2gZnCpzhib5SeWTYX8K1lFODuC+KMIdqm0pQHtaPcmX21/HbLEtiaMx25b3UkYWgFK0oyyMpDK+Ajiv9/CY2/jOnbH6j0LI7QddIj5okyd8wEt4bI7vQV3+rQvveooaNW9Ku8PjPl2WKsPNtUKAX958IJhUq3h66WPxE0U0s+kJfVZNjczhDVq426VPTvSr7fQoQrMvyWfMmb1JUceHLspmZ0F/WQLF0L6P19Zjz7wDdu8ExZvUgyETTqFJl9NYOC+qFteiy5rURQ0xZusffFMgEnT1wGgW/VVMZo9SUi0J92IGuNHNRFNocisKH69RCIZJsShvGwycCKASUWVYDsXEQC3ROzRqxiqh+JECP5/9Ahrj1d+h8lYqvyEKAd0cZ70maTROvlny3YTJJrZsNxUnccgqEFePTDBS27RvBOgCXxli8s3XmJndggip9IorinqA50QxGQ0VVif/ccCHM7ctahd4WVMjQIU2AwAxGUB74LgCyVGB0M9YyDdWdVKB1cyYZyVtBl7B0n0zpsPK8VhZIRJQ2KWtwmt1sLorg3SPDkb PGFUU+w+ PH+qV0q1ViiFriovrlGfd4Zn1/w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: TLB flushes can be skipped if TLB flushes requested have been done by any reason, which doesn't have to be done from migrations. It can be tracked by keeping timestamp(= migrc_gen) when it's requested and when it's triggered. Signed-off-by: Byungchul Park --- arch/x86/include/asm/tlbflush.h | 6 ++++ arch/x86/mm/tlb.c | 55 +++++++++++++++++++++++++++++++++ mm/migrate.c | 10 ++++++ mm/rmap.c | 1 + 4 files changed, 72 insertions(+) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 752d72ea209b..da987c15049e 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -283,6 +283,12 @@ extern void arch_tlbbatch_clean(struct arch_tlbflush_unmap_batch *batch); extern void arch_tlbbatch_fold(struct arch_tlbflush_unmap_batch *bdst, struct arch_tlbflush_unmap_batch *bsrc); +#ifdef CONFIG_MIGRC +extern void arch_migrc_adj(struct arch_tlbflush_unmap_batch *batch, int gen); +#else +static inline void arch_migrc_adj(struct arch_tlbflush_unmap_batch *batch, int gen) {} +#endif + static inline bool pte_flags_need_flush(unsigned long oldflags, unsigned long newflags, bool ignore_access) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 2dabf0f340fb..913cad013979 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1210,9 +1210,48 @@ STATIC_NOPV void native_flush_tlb_local(void) native_write_cr3(__native_read_cr3()); } +#ifdef CONFIG_MIGRC +DEFINE_PER_CPU(int, migrc_done); + +static inline int migrc_tlb_local_begin(void) +{ + int ret = atomic_read(&migrc_gen); + + /* + * XXX: barrier() would be sufficient if the architecture + * quarantees the order between memory access and TLB flush. + */ + smp_mb(); + return ret; +} + +static inline void migrc_tlb_local_end(int gen) +{ + /* + * XXX: barrier() would be sufficient if the architecture + * quarantees the order between TLB flush and memory access. + */ + smp_mb(); + WRITE_ONCE(*this_cpu_ptr(&migrc_done), gen); +} +#else +static inline int migrc_tlb_local_begin(void) +{ + return 0; +} + +static inline void migrc_tlb_local_end(int gen) +{ +} +#endif + void flush_tlb_local(void) { + unsigned int gen; + + gen = migrc_tlb_local_begin(); __flush_tlb_local(); + migrc_tlb_local_end(gen); } /* @@ -1237,6 +1276,22 @@ void __flush_tlb_all(void) } EXPORT_SYMBOL_GPL(__flush_tlb_all); +#ifdef CONFIG_MIGRC +static inline bool before(int a, int b) +{ + return a - b < 0; +} + +void arch_migrc_adj(struct arch_tlbflush_unmap_batch *batch, int gen) +{ + int cpu; + + for_each_cpu(cpu, &batch->cpumask) + if (!before(READ_ONCE(*per_cpu_ptr(&migrc_done, cpu)), gen)) + cpumask_clear_cpu(cpu, &batch->cpumask); +} +#endif + void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { struct flush_tlb_info *info; diff --git a/mm/migrate.c b/mm/migrate.c index f9446f5b312a..c7b72d275b2a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2053,6 +2053,16 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* + * Should be prior to try_to_unmap_flush() so that + * migrc_try_flush_free_folios() that will be called later + * can take benefit from the TLB flushes in try_to_unmap_flush(). + * + * migrc_req_end() will store the timestamp for pending, and + * TLB flushes will also store the timestamp for TLB flush so + * that unnecessary TLB flushes can be skipped using the time + * information. + */ if (migrc_cond1) migrc_req_end(); diff --git a/mm/rmap.c b/mm/rmap.c index 0652d25206ee..2ae1b1324f84 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -627,6 +627,7 @@ static bool __migrc_try_flush_free_folios(struct llist_head *h) llist_for_each_entry_safe(req, req2, reqs, llnode) { struct llist_node *n; + arch_migrc_adj(&req->arch, req->gen); arch_tlbbatch_fold(&arch, &req->arch); n = llist_del_all(&req->pages);