From patchwork Wed Feb 26 12:03:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13992210 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5129C021BF for ; Wed, 26 Feb 2025 12:04:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 135C628003E; Wed, 26 Feb 2025 07:03:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0E79828003D; Wed, 26 Feb 2025 07:03:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA04128003E; Wed, 26 Feb 2025 07:03:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id BF82128003D for ; Wed, 26 Feb 2025 07:03:56 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 85D5651AC2 for ; Wed, 26 Feb 2025 12:03:56 +0000 (UTC) X-FDA: 83161962072.27.6516A09 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf11.hostedemail.com (Postfix) with ESMTP id A99204001D for ; Wed, 26 Feb 2025 12:03:54 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740571435; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=S/vMRtcfMWgcatQ+P8/daAgGR6Fc4eeqIPQa4JLxkOI=; b=rfn2GUg1+7i9HnUSIqiylBosiqYFgo1CTgzoUf93Y75q87oKyhET7Zn+v+ktVkkYM+gs4k ZAochQyv+3gkfJ0Qr1Wm/DP8P8NqIE2j1+02+Z4jtVzZFWjrXFHJVO+8RzaS2+bryGsQnz cdLkTGvnFzIJCZFAzol2EN/dGoDezMw= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740571435; a=rsa-sha256; cv=none; b=unoGiS6WHRykCl3+t0AiwuCjna3g83V9f9kcdVbsHZndsir6KZ1SLap44MZgQVyRoZMfZ6 bdeJtmTtyZPZoxYhqK0EgGFStDlVqpXGtMFNzSlKRWeAfdPc8myzmrsuPxAwAALK32jLz9 khQwOYaJrxaTJ7bMM7B8Kdkxrq3BO/E= X-AuditID: a67dfc5b-3e1ff7000001d7ae-57-67bf03235d70 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, rjgolo@gmail.com Subject: [RFC PATCH v12 based on v6.14-rc4 20/25] mm, fs: skip tlb flushes for luf'd filemap that already has been done Date: Wed, 26 Feb 2025 21:03:31 +0900 Message-Id: <20250226120336.29565-20-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250226120336.29565-1-byungchul@sk.com> References: <20250226113024.GA1935@system.software.com> <20250226120336.29565-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrILMWRmVeSWpSXmKPExsXC9ZZnka4y8/50g31/BCzmrF/DZvF5wz82 i6/rfzFbPP3Ux2JxedccNot7a/6zWpzftZbVYsfSfUwWlw4sYLI43nuAyWL+vc9sFps3TWW2 OD5lKqPF7x9z2Bz4PL639rF47Jx1l91jwaZSj80rtDw2repk89j0aRK7x7tz59g9Tsz4zeLx ft9VNo+tv+w8GqdeY/P4vEkugCeKyyYlNSezLLVI3y6BK2PFfPeCSQEVx+ffZ2pgPOvQxcjB ISFgIrG5XbmLkRPMPLNqMTOIzSagLnHjxk8wW0TATOJg6x/2LkYuDmaBZUwSe080sIEkhAXK JeY13WcDmcMioCrRMJkTJMwLVH//7W5miJnyEqs3HACzOYHin6YdA2sVEkiW2Pn7DxPITAmB 22wSZ1d8gWqQlDi44gbLBEbeBYwMqxiFMvPKchMzc0z0MirzMiv0kvNzNzECA3pZ7Z/oHYyf LgQfYhTgYFTi4X1wZm+6EGtiWXFl7iFGCQ5mJRFezsw96UK8KYmVValF+fFFpTmpxYcYpTlY lMR5jb6VpwgJpCeWpGanphakFsFkmTg4pRoY5V/du9Iu/bKyJq488K5c2IaPKt1vNAz6WXwO qyi7C1g8v/7oGYtLFsfGAKGNk3smxjvf577y6zXPlVr37Bk5IcvdZyRxbhIRdau81vburUtJ tTRfYoFxdvr22kMRJw+tiVU1Ff3jlrUqsbEi+UXj43DNk1rntJ887/ivLHkrnGeiydKskDNK LMUZiYZazEXFiQC5VIXpZAIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrJLMWRmVeSWpSXmKPExsXC5WfdrKvMvD/d4O4zDos569ewWXze8I/N 4uv6X8wWTz/1sVgcnnuS1eLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHG89wCTxfx7n9ks Nm+aymxxfMpURovfP+awOfB7fG/tY/HYOesuu8eCTaUem1doeWxa1cnmsenTJHaPd+fOsXuc mPGbxeP9vqtsHotffGDy2PrLzqNx6jU2j8+b5AJ4o7hsUlJzMstSi/TtErgyVsx3L5gUUHF8 /n2mBsazDl2MnBwSAiYSZ1YtZgax2QTUJW7c+AlmiwiYSRxs/cPexcjFwSywjEli74kGNpCE sEC5xLym+0A2BweLgKpEw2ROkDAvUP39t7uZIWbKS6zecADM5gSKf5p2DKxVSCBZYufvP0wT GLkWMDKsYhTJzCvLTczMMdUrzs6ozMus0EvOz93ECAzQZbV/Ju5g/HLZ/RCjAAejEg/vgzN7 04VYE8uKK3MPMUpwMCuJ8HJm7kkX4k1JrKxKLcqPLyrNSS0+xCjNwaIkzusVnpogJJCeWJKa nZpakFoEk2Xi4JRqYDQ/sezjhw4ej9n/Xil8v/VwjzvPUr6req925djce2K1plTcLvLBg6ec 3Vnfp13LOHf3a23C8YSbea1Lb89fs9x0Eg+rrsSpjGWMsybtniiSqLn1Y9Ueh+z9lzj189a1 Vh5mLi1739+R/kZJ8MeTSX2B7Z0/muqVT/w9Zt/q/uE/V+j8bf1P6syVWIozEg21mIuKEwGV Kex8TAIAAA== X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: A99204001D X-Stat-Signature: jcqbwupydhme5b657za19qhksjm7y6wz X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740571434-397293 X-HE-Meta: U2FsdGVkX18EGHlatCSw+rp0ATAYZWxnCZnxvVqFxWcWDtVqL6qh1BKeE2UVz/PPgL8MJ+TtrP0vldeXCCvRodqVmE8ZYH+EdC/0+BoYKtZ9AG49O+74/+4O6KyRatQmMQAb3gGtJVJfbnsAGLI9jRuv0Ne1uiNdEVAj/ceXYhDDaClXrMcm6K61MiItOLdbM2DdxQZegOex4zLb8UKL7JWhHZJFzYuryhtKMAoZsbZFA95H9UPjyLK2JuAdM2k6Q7fIF9KmXU6xrLLGRFzbd4UqSLNFE9Wk5J4c9Usej/VmoPRvKgxTLrzPX6GrmguE9+FsMz8OiPQse78gHzh7l9JlAzFnkxTVgPm3CZC2ArSqotuAeNOtJwVp3X/LE9HrGwj+Qiads56BGlyO4QCsVPxxxQ6kTwKEeKIelEP2H2TKFnITLja6I91W/mIJ0EZNOpVkHdlycS7h4sgUqKs1KFHrM4Wjd02z+EM74fzj6yqLNn6ciHS5xICpLcykX1O4WgUMeGu2AxI1MXuqYJ3nTgqw2cnbV9vKAXWe0+pn9clPxruE1WR6bHT/Hxek1DkkgaHKpLUuLmHKyKKjooUZZjhBG4KHcVAhcRMbV9BPl5/vX5coR4zSqww7kV3sDyOTeg5HYmWrJn6q2faoYUzbWf7obKUs9sA9BspaeD8GuF8H6cbsa93rSVIlYjeIwDquLvPsJo9K37kHC2uSJsVCVtVGESKOZx19XRvtull+dkzBggrcD8AedM3W2KvomSQmsqA++BPzmwSYCjnKfF4tBGOM01c1L4k1JGspdj8rvFK9+cT9aMVlz3nz0cxHGUHwXT2QTBh9uxCsEKB04BrXc/rQsdVcAQ1V+/XaDie2MUxMxuQlOZfatqBzBYi9zk0DZDfXrd8wPJYy+Q41q/yMMH9ZOB/atp1hdrvx+yJzGF6aKCRm4gwL2EiQxk7Vo8TdrudnJczyDgY9tNZQbnA eow== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For luf'd filemap, tlb shootdown is performed when updating page cache, no matter whether tlb flushes required already has been done or not. By storing luf meta data in struct address_space and updating the luf meta data properly, we can skip unnecessary tlb flush. Signed-off-by: Byungchul Park --- fs/inode.c | 1 + include/linux/fs.h | 4 ++- include/linux/mm_types.h | 2 ++ mm/memory.c | 4 +-- mm/rmap.c | 59 +++++++++++++++++++++++++--------------- mm/truncate.c | 14 +++++----- mm/vmscan.c | 2 +- 7 files changed, 53 insertions(+), 33 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index 5587aabdaa5ee..752fb2df6f3b3 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -475,6 +475,7 @@ static void __address_space_init_once(struct address_space *mapping) init_rwsem(&mapping->i_mmap_rwsem); INIT_LIST_HEAD(&mapping->i_private_list); spin_lock_init(&mapping->i_private_lock); + luf_batch_init(&mapping->luf_batch); mapping->i_mmap = RB_ROOT_CACHED; } diff --git a/include/linux/fs.h b/include/linux/fs.h index 78aaf769d32d1..a2f014b31028f 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -498,6 +498,7 @@ extern const struct address_space_operations empty_aops; * @i_private_lock: For use by the owner of the address_space. * @i_private_list: For use by the owner of the address_space. * @i_private_data: For use by the owner of the address_space. + * @luf_batch: Data to track need of tlb flush by luf. */ struct address_space { struct inode *host; @@ -519,6 +520,7 @@ struct address_space { struct list_head i_private_list; struct rw_semaphore i_mmap_rwsem; void * i_private_data; + struct luf_batch luf_batch; } __attribute__((aligned(sizeof(long)))) __randomize_layout; /* * On most architectures that alignment is already the case; but @@ -545,7 +547,7 @@ static inline int mapping_write_begin(struct file *file, * Ensure to clean stale tlb entries for this mapping. */ if (!ret) - luf_flush(0); + luf_flush_mapping(mapping); return ret; } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 2ac93d4f67c15..96015fc68e4f5 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1348,10 +1348,12 @@ extern void tlb_finish_mmu(struct mmu_gather *tlb); void luf_flush(unsigned short luf_key); void luf_flush_mm(struct mm_struct *mm); void luf_flush_vma(struct vm_area_struct *vma); +void luf_flush_mapping(struct address_space *mapping); #else static inline void luf_flush(unsigned short luf_key) {} static inline void luf_flush_mm(struct mm_struct *mm) {} static inline void luf_flush_vma(struct vm_area_struct *vma) {} +static inline void luf_flush_mapping(struct address_space *mapping) {} #endif struct vm_fault; diff --git a/mm/memory.c b/mm/memory.c index 52bd45fe00f55..6cdc1df0424f3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -6261,10 +6261,10 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *vma, unsigned long address, if (flush) { /* * If it has a VM_SHARED mapping, all the mms involved - * should be luf_flush'ed. + * in the struct address_space should be luf_flush'ed. */ if (mapping) - luf_flush(0); + luf_flush_mapping(mapping); luf_flush_mm(mm); } diff --git a/mm/rmap.c b/mm/rmap.c index d68cfd28e0939..58dfc9889b1ee 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -691,7 +691,7 @@ void fold_batch(struct tlbflush_unmap_batch *dst, #define NR_LUF_BATCH (1 << (sizeof(short) * 8)) /* - * Use 0th entry as accumulated batch. + * XXX: Reserve the 0th entry for later use. */ struct luf_batch luf_batch[NR_LUF_BATCH]; @@ -936,7 +936,7 @@ void luf_flush_vma(struct vm_area_struct *vma) mapping = vma->vm_file->f_mapping; if (mapping) - luf_flush(0); + luf_flush_mapping(mapping); luf_flush_mm(mm); } @@ -962,6 +962,29 @@ void luf_flush_mm(struct mm_struct *mm) try_to_unmap_flush(); } +void luf_flush_mapping(struct address_space *mapping) +{ + struct tlbflush_unmap_batch *tlb_ubc = ¤t->tlb_ubc; + struct luf_batch *lb; + unsigned long flags; + unsigned long lb_ugen; + + if (!mapping) + return; + + lb = &mapping->luf_batch; + read_lock_irqsave(&lb->lock, flags); + fold_batch(tlb_ubc, &lb->batch, false); + lb_ugen = lb->ugen; + read_unlock_irqrestore(&lb->lock, flags); + + if (arch_tlbbatch_diet(&tlb_ubc->arch, lb_ugen)) + return; + + try_to_unmap_flush(); +} +EXPORT_SYMBOL(luf_flush_mapping); + /* * Flush TLB entries for recently unmapped pages from remote CPUs. It is * important if a PTE was dirty when it was unmapped that it's flushed @@ -1010,7 +1033,8 @@ void try_to_unmap_flush_dirty(void) static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, unsigned long uaddr, - struct vm_area_struct *vma) + struct vm_area_struct *vma, + struct address_space *mapping) { struct tlbflush_unmap_batch *tlb_ubc; int batch; @@ -1032,27 +1056,15 @@ static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, tlb_ubc = ¤t->tlb_ubc; else { tlb_ubc = ¤t->tlb_ubc_ro; + fold_luf_batch_mm(&mm->luf_batch, mm); + if (mapping) + fold_luf_batch_mm(&mapping->luf_batch, mm); } arch_tlbbatch_add_pending(&tlb_ubc->arch, mm, uaddr); tlb_ubc->flush_required = true; - if (can_luf_test()) { - struct luf_batch *lb; - unsigned long flags; - - /* - * Accumulate to the 0th entry right away so that - * luf_flush(0) can be uesed to properly perform pending - * TLB flush once this unmapping is observed. - */ - lb = &luf_batch[0]; - write_lock_irqsave(&lb->lock, flags); - __fold_luf_batch(lb, tlb_ubc, new_luf_ugen()); - write_unlock_irqrestore(&lb->lock, flags); - } - /* * Ensure compiler does not re-order the setting of tlb_flush_batched * before the PTE is cleared. @@ -1134,7 +1146,8 @@ void flush_tlb_batched_pending(struct mm_struct *mm) #else static void set_tlb_ubc_flush_pending(struct mm_struct *mm, pte_t pteval, unsigned long uaddr, - struct vm_area_struct *vma) + struct vm_area_struct *vma, + struct address_space *mapping) { } @@ -1503,7 +1516,7 @@ int folio_mkclean(struct folio *folio) /* * Ensure to clean stale tlb entries for this mapping. */ - luf_flush(0); + luf_flush_mapping(mapping); return cleaned; } @@ -2037,6 +2050,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, enum ttu_flags flags = (enum ttu_flags)(long)arg; unsigned long pfn; unsigned long hsz = 0; + struct address_space *mapping = folio_mapping(folio); /* * When racing against e.g. zap_pte_range() on another cpu, @@ -2174,7 +2188,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pteval, address, vma); + set_tlb_ubc_flush_pending(mm, pteval, address, vma, mapping); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } @@ -2414,6 +2428,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, enum ttu_flags flags = (enum ttu_flags)(long)arg; unsigned long pfn; unsigned long hsz = 0; + struct address_space *mapping = folio_mapping(folio); /* * When racing against e.g. zap_pte_range() on another cpu, @@ -2563,7 +2578,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ pteval = ptep_get_and_clear(mm, address, pvmw.pte); - set_tlb_ubc_flush_pending(mm, pteval, address, vma); + set_tlb_ubc_flush_pending(mm, pteval, address, vma, mapping); } else { pteval = ptep_clear_flush(vma, address, pvmw.pte); } diff --git a/mm/truncate.c b/mm/truncate.c index 2bf3806391c21..b2934c4edebf1 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -128,7 +128,7 @@ void folio_invalidate(struct folio *folio, size_t offset, size_t length) /* * Ensure to clean stale tlb entries for this mapping. */ - luf_flush(0); + luf_flush_mapping(folio->mapping); } EXPORT_SYMBOL_GPL(folio_invalidate); @@ -169,7 +169,7 @@ int truncate_inode_folio(struct address_space *mapping, struct folio *folio) /* * Ensure to clean stale tlb entries for this mapping. */ - luf_flush(0); + luf_flush_mapping(mapping); return 0; } @@ -219,7 +219,7 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) /* * Ensure to clean stale tlb entries for this mapping. */ - luf_flush(0); + luf_flush_mapping(folio->mapping); if (!folio_test_large(folio)) return true; @@ -281,7 +281,7 @@ long mapping_evict_folio(struct address_space *mapping, struct folio *folio) /* * Ensure to clean stale tlb entries for this mapping. */ - luf_flush(0); + luf_flush_mapping(mapping); return ret; } @@ -416,7 +416,7 @@ void truncate_inode_pages_range(struct address_space *mapping, /* * Ensure to clean stale tlb entries for this mapping. */ - luf_flush(0); + luf_flush_mapping(mapping); } EXPORT_SYMBOL(truncate_inode_pages_range); @@ -536,7 +536,7 @@ unsigned long mapping_try_invalidate(struct address_space *mapping, /* * Ensure to clean stale tlb entries for this mapping. */ - luf_flush(0); + luf_flush_mapping(mapping); return count; } @@ -706,7 +706,7 @@ int invalidate_inode_pages2_range(struct address_space *mapping, /* * Ensure to clean stale tlb entries for this mapping. */ - luf_flush(0); + luf_flush_mapping(mapping); return ret; } EXPORT_SYMBOL_GPL(invalidate_inode_pages2_range); diff --git a/mm/vmscan.c b/mm/vmscan.c index 461e7643898e7..a31a7cf87315f 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -843,7 +843,7 @@ long remove_mapping(struct address_space *mapping, struct folio *folio) /* * Ensure to clean stale tlb entries for this mapping. */ - luf_flush(0); + luf_flush_mapping(mapping); return ret; }