From patchwork Wed Aug 30 09:50:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13370145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E154C6FA8F for ; Wed, 30 Aug 2023 09:50:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 048A3280050; Wed, 30 Aug 2023 05:50:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F13A628004E; Wed, 30 Aug 2023 05:50:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D8CF3280050; Wed, 30 Aug 2023 05:50:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C84CC28004E for ; Wed, 30 Aug 2023 05:50:39 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8E8F6120173 for ; Wed, 30 Aug 2023 09:50:39 +0000 (UTC) X-FDA: 81180301398.25.D90BD8B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id C963A140004 for ; Wed, 30 Aug 2023 09:50:37 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693389037; a=rsa-sha256; cv=none; b=SgEsTn/DvF5naDxu8V3PGq8XW8cjyA5UAp2CLNWQ6Ba2fOOJkLIZ+IyV71Ot7lLI5nPfAF gqmAoDKLb5n0C+PC0tHTGQO5GqNRv7RJlkAmRKbPjMK5XT+G6Hdi0BJ7uGjnc+tQTjQJMd H0jn76JX2FLfnlkT8l+dz5iBK28/VIA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693389037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wmrVLSJPa0RjCTuS+lqhx0P0ZiJJRUlJnABwX77Ults=; b=ERxVOv+QyQ0FGyTlxvl4dSoscOoGZ1QirnEKqwx0nKayTA6Y/v4ZpP2dBL5KQ4djz5MPVn Z5Aeg/cFvtVS10ISQNllwT3/fBh8J9EvMx0luqjsWeOla/Ca37uwystZPeYKSh7gkxHZmX MtmKXVuDALdxsV1eo0O4ufrloR3Ljz4= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 646692F4; Wed, 30 Aug 2023 02:51:16 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 659443F64C; Wed, 30 Aug 2023 02:50:34 -0700 (PDT) From: Ryan Roberts To: Will Deacon , "Aneesh Kumar K.V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yu Zhao , "Kirill A. Shutemov" , Yin Fengwei , Yang Shi , "Huang, Ying" , Zi Yan Cc: Ryan Roberts , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/5] mm/mmu_gather: Remove encoded_page infrastructure Date: Wed, 30 Aug 2023 10:50:09 +0100 Message-Id: <20230830095011.1228673-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20230830095011.1228673-1-ryan.roberts@arm.com> References: <20230830095011.1228673-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: C963A140004 X-Stat-Signature: tw4iwkgqqtfi941qwhmu6bmdm4s7hmo8 X-Rspam-User: X-HE-Tag: 1693389037-351918 X-HE-Meta: U2FsdGVkX18qqW4DRhBvzr5j2kjqRWiXUuCBb7WVwOnwhzrzhgFNhfuYMjWFflOB6G/rJ9ck/UoIqkYy9Mc2HJlJ8aS2IF/6hmjMnftpIPaNoYnYMcUSsDEhdX8MQWkVXm8tj1ZSHdM+TqfRbxrlaIn+b8mWo3VMFnRizQ5GFW6ZgqtOko3+9RRaXtLZnvYd3lQs1eyXhnYTMIymnGZOEvwK+1VOu/9HXEaRut8ckmC8pPVyfV1RTxI4x+L/6dANg5vsn4GY4pqtu0XrwjPhx76Oqi+LJOU3QdHUAcYxZJxxXQS7Z8qcPiXNQFFOqhQ/COf5eYMhpusZtofqqYrJluNkDp7jP9K85KwNzreNGUAUfwQhC5J+Oczr3EWOCdEhtytVplDvN5lR/YtBn6D7YiV3dL/2dXQ/guev2CPFvLXDgGKqKzqRH4cSByB4Fea2VoCgDIsct5vn7IQF7E26Olw2Dk0xGQHS+d8HLt2YNKMq12/JGOvHPT/BC+BzsF5zDd4dOmEd/2kKBW6QQiXARMcysVc6TY3HYcYWipo1qf2xSyg9fCglQCJGrRGYIRB9+xVHcI7O9hOu4NdpsD6quPGSuqXhzs1ZWTrwuAH9BDehConvbQbzdJfh+hfCi16CrmawrqvtSRLXtAKjvkFr2J1GsPqbZCjnwnVYgLr6HoDJFbdblZD+7L4nYh/cD9iH51UYkzCCzbvxU7oOeDQVr9+60O2tVbtWpmEc7mAnG6rNBpIiz58C7GFlW8Kt1bbIQntndHKzb5dKLQHzyNKkb0YvoYyRRgzw7gpSxYpGkKsH48utQsqq1id8bhrcdRCi5ybhK06AHbW68AlQRz+EPfvj1gOhddeOjJfLlmYjcjPh+pAyQY2UJvDGamf1MuNJoefiqI/zFHLCO6ox+K1r8Vn6/Xrhb4s7d60/qftGhMCPkg1Ruunuf/WpVxZ1yzuNAmS6ja8dOno1/pMdsTk ozyjutUv /AibESeO7NbBhHKd/csGC2GcJ3KaveCJjBVmwSzQf5+Eaa7ocUHECboNMEt8NZemqaZgx5zc/V+ZwdWAivpT1OBcRno4f7SuKNbU4MOFOIlbFMojz8RxA0NeHGK8Lj1U1andIhblheviQ6+XsOp5OSK0JN5HUGA/dUaxsLel1HzEVDKhfGLt5nGeu0ue/eissmPUWlUgaY/CJJRg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: commit 70fb4fdff582 ("mm: introduce 'encoded' page pointers with embedded extra bits") and commit 7cc8f9c7146a ("mm: mmu_gather: prepare to gather encoded page pointers with flags") converted mmu_gather for dealing with encoded_page, where the bottom 2 bits could encode extra flags. Only 1 bit was ever used; to flag whether the page should participate in a delayed rmap removal. Now that the mmu_gather batched rmap removal mechanism has been generalized, all pages participate and therefore the flag is unused. So let's remove encoded_page to simplify the code. It also gets in the way of further optimization which will be done in a follow up patch. Signed-off-by: Ryan Roberts --- arch/s390/include/asm/tlb.h | 9 +++------ include/asm-generic/tlb.h | 10 +++++----- include/linux/mm.h | 4 +--- include/linux/mm_types.h | 34 +--------------------------------- include/linux/swap.h | 2 +- mm/memory.c | 2 +- mm/mmu_gather.c | 11 +++++------ mm/swap.c | 8 +++----- mm/swap_state.c | 4 ++-- 9 files changed, 22 insertions(+), 62 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index 383b1f91442c..c40b44f6a31b 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -25,7 +25,7 @@ void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, + struct page *page, int page_size); #define tlb_flush tlb_flush @@ -41,15 +41,12 @@ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, * Release the page cache reference for a pte removed by * tlb_ptep_clear_flush. In both flush modes the tlb for a page cache page * has already been freed, so just do free_page_and_swap_cache. - * - * s390 doesn't delay rmap removal, so there is nothing encoded in - * the page pointer. */ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, + struct page *page, int page_size) { - free_page_and_swap_cache(encoded_page_ptr(page)); + free_page_and_swap_cache(page); return false; } diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index f339d68cf44f..d874415aaa33 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -246,7 +246,7 @@ struct mmu_gather_batch { struct mmu_gather_batch *next; unsigned int nr; unsigned int max; - struct encoded_page *encoded_pages[]; + struct page *pages[]; }; #define MAX_GATHER_BATCH \ @@ -261,7 +261,7 @@ struct mmu_gather_batch { #define MAX_GATHER_BATCH_COUNT (10000UL/MAX_GATHER_BATCH) extern bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, + struct page *page, int page_size); #ifdef CONFIG_SMP @@ -464,13 +464,13 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) static inline void tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) { - if (__tlb_remove_page_size(tlb, encode_page(page, 0), page_size)) + if (__tlb_remove_page_size(tlb, page, page_size)) tlb_flush_mmu(tlb); } -static __always_inline bool __tlb_remove_page(struct mmu_gather *tlb, struct page *page, unsigned int flags) +static __always_inline bool __tlb_remove_page(struct mmu_gather *tlb, struct page *page) { - return __tlb_remove_page_size(tlb, encode_page(page, flags), PAGE_SIZE); + return __tlb_remove_page_size(tlb, page, PAGE_SIZE); } /* tlb_remove_page diff --git a/include/linux/mm.h b/include/linux/mm.h index 53efddc4d178..9cd20a38089c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1481,8 +1481,7 @@ static inline void folio_put_refs(struct folio *folio, int refs) * * release_pages() releases a simple array of multiple pages, and * accepts various different forms of said page array: either - * a regular old boring array of pages, an array of folios, or - * an array of encoded page pointers. + * a regular old boring array of pages or an array of folios. * * The transparent union syntax for this kind of "any of these * argument types" is all kinds of ugly, so look away. @@ -1490,7 +1489,6 @@ static inline void folio_put_refs(struct folio *folio, int refs) typedef union { struct page **pages; struct folio **folios; - struct encoded_page **encoded_pages; } release_pages_arg __attribute__ ((__transparent_union__)); void release_pages(release_pages_arg, int nr); diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 5a995089cbf5..e9a0daf0c8d4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -68,7 +68,7 @@ struct mem_cgroup; #ifdef CONFIG_HAVE_ALIGNED_STRUCT_PAGE #define _struct_page_alignment __aligned(2 * sizeof(unsigned long)) #else -#define _struct_page_alignment __aligned(sizeof(unsigned long)) +#define _struct_page_alignment #endif struct page { @@ -216,38 +216,6 @@ struct page { #endif } _struct_page_alignment; -/* - * struct encoded_page - a nonexistent type marking this pointer - * - * An 'encoded_page' pointer is a pointer to a regular 'struct page', but - * with the low bits of the pointer indicating extra context-dependent - * information. Not super-common, but happens in mmu_gather and mlock - * handling, and this acts as a type system check on that use. - * - * We only really have two guaranteed bits in general, although you could - * play with 'struct page' alignment (see CONFIG_HAVE_ALIGNED_STRUCT_PAGE) - * for more. - * - * Use the supplied helper functions to endcode/decode the pointer and bits. - */ -struct encoded_page; -#define ENCODE_PAGE_BITS 3ul -static __always_inline struct encoded_page *encode_page(struct page *page, unsigned long flags) -{ - BUILD_BUG_ON(flags > ENCODE_PAGE_BITS); - return (struct encoded_page *)(flags | (unsigned long)page); -} - -static inline unsigned long encoded_page_flags(struct encoded_page *page) -{ - return ENCODE_PAGE_BITS & (unsigned long)page; -} - -static inline struct page *encoded_page_ptr(struct encoded_page *page) -{ - return (struct page *)(~ENCODE_PAGE_BITS & (unsigned long)page); -} - /* * A swap entry has to fit into a "unsigned long", as the entry is hidden * in the "index" field of the swapper address space. diff --git a/include/linux/swap.h b/include/linux/swap.h index 493487ed7c38..9e12c6d49997 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -451,7 +451,7 @@ static inline unsigned long total_swapcache_pages(void) extern void free_swap_cache(struct page *page); extern void free_page_and_swap_cache(struct page *); -extern void free_pages_and_swap_cache(struct encoded_page **, int); +extern void free_pages_and_swap_cache(struct page **, int); /* linux/mm/swapfile.c */ extern atomic_long_t nr_swap_pages; extern long total_swap_pages; diff --git a/mm/memory.c b/mm/memory.c index 823c8a6813d1..3d5d395caba4 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1460,7 +1460,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } - if (unlikely(__tlb_remove_page(tlb, page, 0))) { + if (unlikely(__tlb_remove_page(tlb, page))) { force_flush = 1; addr += PAGE_SIZE; break; diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index fb34151c0da9..cdebb5b9f5c4 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -49,8 +49,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_struct *vma) { for (int i = first; i < batch->nr; i++) { - struct encoded_page *enc = batch->encoded_pages[i]; - struct page *page = encoded_page_ptr(enc); + struct page *page = batch->pages[i]; page_remove_rmap(page, vma, false); } @@ -95,7 +94,7 @@ static void tlb_batch_pages_flush(struct mmu_gather *tlb) struct mmu_gather_batch *batch; for (batch = &tlb->local; batch && batch->nr; batch = batch->next) { - struct encoded_page **pages = batch->encoded_pages; + struct page **pages = batch->pages; do { /* @@ -125,7 +124,7 @@ static void tlb_batch_list_free(struct mmu_gather *tlb) tlb->local.next = NULL; } -bool __tlb_remove_page_size(struct mmu_gather *tlb, struct encoded_page *page, int page_size) +bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) { struct mmu_gather_batch *batch; @@ -140,13 +139,13 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct encoded_page *page, i * Add the page and check if we are full. If so * force a flush. */ - batch->encoded_pages[batch->nr++] = page; + batch->pages[batch->nr++] = page; if (batch->nr == batch->max) { if (!tlb_next_batch(tlb)) return true; batch = tlb->active; } - VM_BUG_ON_PAGE(batch->nr > batch->max, encoded_page_ptr(page)); + VM_BUG_ON_PAGE(batch->nr > batch->max, page); return false; } diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..b05cce475202 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -953,14 +953,12 @@ void lru_cache_disable(void) * Decrement the reference count on all the pages in @arg. If it * fell to zero, remove the page from the LRU and free it. * - * Note that the argument can be an array of pages, encoded pages, - * or folio pointers. We ignore any encoded bits, and turn any of - * them into just a folio that gets free'd. + * Note that the argument can be an array of pages or folio pointers. */ void release_pages(release_pages_arg arg, int nr) { int i; - struct encoded_page **encoded = arg.encoded_pages; + struct page **pages = arg.pages; LIST_HEAD(pages_to_free); struct lruvec *lruvec = NULL; unsigned long flags = 0; @@ -970,7 +968,7 @@ void release_pages(release_pages_arg arg, int nr) struct folio *folio; /* Turn any of the argument types into a folio */ - folio = page_folio(encoded_page_ptr(encoded[i])); + folio = page_folio(pages[i]); /* * Make sure the IRQ-safe lock-holding time does not get diff --git a/mm/swap_state.c b/mm/swap_state.c index b3b14bd0dd64..2132340c6e61 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -307,11 +307,11 @@ void free_page_and_swap_cache(struct page *page) * Passed an array of pages, drop them all from swapcache and then release * them. They are removed from the LRU and freed if this is their last use. */ -void free_pages_and_swap_cache(struct encoded_page **pages, int nr) +void free_pages_and_swap_cache(struct page **pages, int nr) { lru_add_drain(); for (int i = 0; i < nr; i++) - free_swap_cache(encoded_page_ptr(pages[i])); + free_swap_cache(pages[i]); release_pages(pages, nr); }