From patchwork Wed Mar 8 09:41:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13165511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92BA3C64EC4 for ; Wed, 8 Mar 2023 09:41:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 341006B0075; Wed, 8 Mar 2023 04:41:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F06A6B0078; Wed, 8 Mar 2023 04:41:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1919A280001; Wed, 8 Mar 2023 04:41:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 08F436B0075 for ; Wed, 8 Mar 2023 04:41:33 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D148C1C676E for ; Wed, 8 Mar 2023 09:41:32 +0000 (UTC) X-FDA: 80545238424.12.97B6E68 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf13.hostedemail.com (Postfix) with ESMTP id 89C1C20012 for ; Wed, 8 Mar 2023 09:41:30 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=g2gtD1xh; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of rppt@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678268491; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7mWerlB7iI5JwLACExHKsBgwf4sqipB5x9+UyIghBDI=; b=452JR5Gko813YEVAopLFZDgaIgVqCrI6YCVbxtcHc0+YK9JWBCUVvJgq6jAj667jPNiBsk pHQDCzpMhr6ib5vl13/TkmKeZ1oTSalxmDlgnBpcsX57itt4mdVe48dRc8BuHdd+R/QLVl i2+A/dDM7jGJx79aVIc6UW+M4HAoN5U= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=g2gtD1xh; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of rppt@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=rppt@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678268491; a=rsa-sha256; cv=none; b=WPrGd/Mo5lI9wC9H5i55/j5Khnuqhne4vK4LXjIQvwcINpZNusoXxr/OWS0xkgTrMeT5xz sQG1s70nOjnzvpFd2lUAuoqOwDxNbkTGbPtehflFyNxnWTpWFkEmXKtqgNfdhB726okMgC nz+/xns8zjw01M1Es6m5QSZ7fueBdb4= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id 58B70CE1EF8; Wed, 8 Mar 2023 09:41:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2E1AAC4339B; Wed, 8 Mar 2023 09:41:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268484; bh=yWy9YBr3QRwlwUhGDvVMLg0rTuhOsa1avYZumFur5Fc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g2gtD1xhW3u8pPRbZe/MH1SYrnQyB3Pg3cb3uhQHhN6UnNrqhop4CTR6FmDAcS2St YCjcuOxw+R2wN9J/DVCCiwFM1JQcWLMMdg1T7gVK2z2YpXDqJruWy/AlgmPjaCy147 JGJvvZPr9xvET4zf6lp6RmFzW7BMjIPm/A2p+scS/WAQF9qM78NbN+qaLFs0wooc6t H+Ixc0EkDbJU229EW9sLUolO7nPNsXja4Aq4l/mDMesoTliZWImHdvrCLOpMy+NyMW VntaGV1oKqnqu8ipRu6ukpj4mvrmJREPGAWAm85CMkMG5ERTOgugoBXvyVkB9UWYpg bkJ8fmxQCrpmg== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 1/5] mm: intorduce __GFP_UNMAPPED and unmapped_alloc() Date: Wed, 8 Mar 2023 11:41:02 +0200 Message-Id: <20230308094106.227365-2-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 89C1C20012 X-Stat-Signature: 8k16ppfton1w43rauqispiwf555yc4sw X-HE-Tag: 1678268490-226872 X-HE-Meta: U2FsdGVkX1/eQgHMt+YfeysYG9dSzXSKLcFEprbulco+tZpEMonLErW4ET5ldg7ut12gw3sSJ/jvUar5xo+QWISfQ2jo8ACtRBPKqsU05y6VcGrBiJqff3xs+5DJfqDapJHd2kp8DARWEjMez3fBJs/wztCfe0dChcc4uW7bZ1YssKAnJg7h117V/F3Fb7x+pdm4PG/oeOsvR9Jf056kp9CZ9NZO5VKkFtdDPUurIZJawlWMPLz05L5eeg7Wx/Fi2rn2+t1ssZUGVMxiMm6gT0sx55nXsfJcFYaCYIPH3eiNk7ssc2qYC1J4ppQembBf0KYB8IRjCfSeRpv+6Ar5WLTZFaycSqB/GK4XCNWzRvmA5y/xm0HvhpYfyJpXa6D1kaAX5ghDA2xC6rsbVgh2NotLskqAxj7wmGW8uk3s5CZmIMKM6VhOVVhlVJnDFWY8U7UET5TXrgHTQAT8mJTfUFQJVKo9E+yReYaHuaAt3NM94JnqCI+BtGkm/kgyU/T8LjnRZlvg+KBWr36oSGc2gXkatXV5S+g+/rHk3OBLPhHrjUIHAuLnTcA0z4dy2uiEmdxYJqhgoX0K9gwqvLsn47m3zgSpDscctHvQJUPEUkH+wJvTGyQH/HuxJ30a6bjjyVL50E2Wq/2HJTrGgkxHLhSeqqG/4sOfcXUUCrphAVWsd4di8uP7bvYUUYpj7N0SjbTnDhfv6cnrw17rocHwa1yrOMPz4oEPbs7cJliR13ururrPlAskATpF7SvlEXnvDUNdEiQS78TWNd9r0ofMhDWlI9DnYyHRZLd+P85UKCXqj6hl2vstIZf6XF4RMYdnT3QABTmc8V5TCQfloirxn+dI39lTsiowNu2sPGGG66XmP+zySl3JejoPrfpX9zdeM/+McE8GEEafxluJkf/DRB/uSoiP0S/590f0hRwuwLog6ANnPbR+gIv+V7juTBheg4rpoZXcwz5mm6G+zKX e02npPr3 ocmgiw/j6OHAHYqufjkWdTEZ57qNczJIwKfYapefH7THAWECDDrLGFvH7aNdbnX/jP6Jts1P7aau7Fa844Nhwqz0ReIRMlVSGf15cETuPp+ATU7HQlHKLAKGd2eBsJzHPYp9x2cnj5fZorLdEqGSmthqDq8q61XWjNRAbCf2MPTZKw+BgkQmasw64P/TcWTruwVtb3ciUzHnogxNV7/g98flvqJAZFKX674q8lQZ5E8TERwEFm9vthneraxjJzyYT8CvjTBl3ejvkuS1qgGb4go3PdGdgQe0MdxCwvck72dUnpyN1zf4lPXjSWhBYwydr05WUShhFhtXPA9wjlWO4+nQs4EDXWNkpqyaDrXe4jS/N9RyMv7tQ1tdRbXh6Xj4rbylowlOz/3WHST9s4GhzmKiz8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" When set_memory or set_direct_map APIs used to change attribute or permissions for chunks of several pages, the large PMD that maps these pages in the direct map must be split. Fragmenting the direct map in such manner causes TLB pressure and, eventually, performance degradation. To avoid excessive direct map fragmentation, add ability to allocate "unmapped" pages with __GFP_UNMAPPED flag that will cause removal of the allocated pages from the direct map and use a cache of the unmapped pages. This cache is replenished with higher order pages with preference for PMD_SIZE pages when possible so that there will be fewer splits of large pages in the direct map. The cache is implemented as a buddy allocator, so it can serve high order allocations of unmapped pages. Signed-off-by: Mike Rapoport (IBM) Signed-off-by: Kent Overstreet --- arch/x86/Kconfig | 3 + include/linux/gfp_types.h | 11 +- include/linux/page-flags.h | 6 + include/linux/pageblock-flags.h | 28 +++++ include/trace/events/mmflags.h | 10 +- mm/Kconfig | 4 + mm/Makefile | 1 + mm/internal.h | 22 ++++ mm/page_alloc.c | 29 ++++- mm/unmapped-alloc.c | 215 ++++++++++++++++++++++++++++++++ 10 files changed, 325 insertions(+), 4 deletions(-) create mode 100644 mm/unmapped-alloc.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index a825bf031f49..735f691d449c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -35,6 +35,9 @@ config X86_64 select ARCH_HAS_ELFCORE_COMPAT select ZONE_DMA32 +config ARCH_WANTS_GFP_UNMAPPED + def_bool y if X86_64 + config FORCE_DYNAMIC_FTRACE def_bool y depends on X86_32 diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 5088637fe5c2..234122b434dd 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -60,6 +60,12 @@ typedef unsigned int __bitwise gfp_t; #else #define ___GFP_NOLOCKDEP 0 #endif +#ifdef CONFIG_UNMAPPED_ALLOC +#define ___GFP_UNMAPPED 0x10000000u +#else +#define ___GFP_UNMAPPED 0 +#endif + /* If the above are modified, __GFP_BITS_SHIFT may need updating */ /* @@ -101,12 +107,15 @@ typedef unsigned int __bitwise gfp_t; * node with no fallbacks or placement policy enforcements. * * %__GFP_ACCOUNT causes the allocation to be accounted to kmemcg. + * + * %__GFP_UNMAPPED removes the allocated pages from the direct map. */ #define __GFP_RECLAIMABLE ((__force gfp_t)___GFP_RECLAIMABLE) #define __GFP_WRITE ((__force gfp_t)___GFP_WRITE) #define __GFP_HARDWALL ((__force gfp_t)___GFP_HARDWALL) #define __GFP_THISNODE ((__force gfp_t)___GFP_THISNODE) #define __GFP_ACCOUNT ((__force gfp_t)___GFP_ACCOUNT) +#define __GFP_UNMAPPED ((__force gfp_t)___GFP_UNMAPPED) /** * DOC: Watermark modifiers @@ -252,7 +261,7 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (28 + IS_ENABLED(CONFIG_LOCKDEP)) #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a7e3a3405520..e66dbfdba8f2 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -922,6 +922,7 @@ static inline bool is_page_hwpoison(struct page *page) #define PG_offline 0x00000100 #define PG_table 0x00000200 #define PG_guard 0x00000400 +#define PG_unmapped 0x00000800 #define PageType(page, flag) \ ((page->page_type & (PAGE_TYPE_BASE | flag)) == PAGE_TYPE_BASE) @@ -992,6 +993,11 @@ PAGE_TYPE_OPS(Table, table) */ PAGE_TYPE_OPS(Guard, guard) +/* + * Marks pages in free lists of unmapped allocator. + */ +PAGE_TYPE_OPS(Unmapped, unmapped) + extern bool is_free_buddy_page(struct page *page); PAGEFLAG(Isolated, isolated, PF_ANY); diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flags.h index 5f1ae07d724b..bb6f5ec83881 100644 --- a/include/linux/pageblock-flags.h +++ b/include/linux/pageblock-flags.h @@ -21,6 +21,9 @@ enum pageblock_bits { /* 3 bits required for migrate types */ PB_migrate_skip,/* If set the block is skipped by compaction */ + PB_unmapped = 7,/* if set the block has pages unmapped in the direct + map */ + /* * Assume the bits will always align on a word. If this assumption * changes then get/set pageblock needs updating. @@ -95,4 +98,29 @@ static inline void set_pageblock_skip(struct page *page) } #endif /* CONFIG_COMPACTION */ +#ifdef CONFIG_UNMAPPED_ALLOC +#define get_pageblock_unmapped(page) \ + get_pfnblock_flags_mask(page, page_to_pfn(page), \ + (1 << (PB_unmapped))) +#define clear_pageblock_unmapped(page) \ + set_pfnblock_flags_mask(page, 0, page_to_pfn(page), \ + (1 << PB_unmapped)) +#define set_pageblock_unmapped(page) \ + set_pfnblock_flags_mask(page, (1 << PB_unmapped), \ + page_to_pfn(page), \ + (1 << PB_unmapped)) +#else /* CONFIG_UNMAPPED_ALLOC */ +static inline bool get_pageblock_unmapped(struct page *page) +{ + return false; +} +static inline void clear_pageblock_unmapped(struct page *page) +{ +} +static inline void set_pageblock_unmapped(struct page *page) +{ +} +#endif /* CONFIG_UNMAPPED_ALLOC */ + + #endif /* PAGEBLOCK_FLAGS_H */ diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 9db52bc4ce19..951d294a3840 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -61,9 +61,17 @@ #define __def_gfpflag_names_kasan #endif +#ifdef CONFIG_UNMAPPED_ALLOC +#define __def_gfpflag_names_unmapped \ + , gfpflag_string(__GFP_UNMAPPED) +#else +#define __def_gfpflag_names_unmapped +#endif + #define show_gfp_flags(flags) \ (flags) ? __print_flags(flags, "|", \ - __def_gfpflag_names __def_gfpflag_names_kasan \ + __def_gfpflag_names __def_gfpflag_names_kasan \ + __def_gfpflag_names_unmapped \ ) : "none" #ifdef CONFIG_MMU diff --git a/mm/Kconfig b/mm/Kconfig index 4751031f3f05..404be73e00e8 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1202,6 +1202,10 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config UNMAPPED_ALLOC + def_bool y + depends on ARCH_WANTS_GFP_UNMAPPED + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index 8e105e5b3e29..9143303295af 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -138,3 +138,4 @@ obj-$(CONFIG_IO_MAPPING) += io-mapping.o obj-$(CONFIG_HAVE_BOOTMEM_INFO_NODE) += bootmem_info.o obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o +obj-$(CONFIG_UNMAPPED_ALLOC) += unmapped-alloc.o diff --git a/mm/internal.h b/mm/internal.h index 7920a8b7982e..8d84cceab467 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1042,4 +1042,26 @@ struct vma_prepare { struct vm_area_struct *remove; struct vm_area_struct *remove2; }; + +/* + * mm/unmapped-alloc.c + */ +#ifdef CONFIG_UNMAPPED_ALLOC +int unmapped_alloc_init(void); +struct page *unmapped_pages_alloc(gfp_t gfp, int order); +void unmapped_pages_free(struct page *page, int order); +#else +static inline int unmapped_alloc_init(void) +{ + return 0; +} + +static inline struct page *unmapped_pages_alloc(gfp_t gfp, int order) +{ + return NULL; +} + +static inline void unmapped_pages_free(struct page *page, int order) {} +#endif + #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ac1fc986af44..01f18e7529b0 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -589,7 +589,7 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags, unsigned long bitidx, word_bitidx; unsigned long word; - BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 4); + BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 8); BUILD_BUG_ON(MIGRATE_TYPES > (1 << PB_migratetype_bits)); bitmap = get_pageblock_bitmap(page, pfn); @@ -746,6 +746,13 @@ static inline bool pcp_allowed_order(unsigned int order) return false; } +static inline bool pcp_allowed(unsigned int order, gfp_t gfp) +{ + if (unlikely(gfp & __GFP_UNMAPPED)) + return false; + return pcp_allowed_order(order); +} + static inline void free_the_page(struct page *page, unsigned int order) { if (pcp_allowed_order(order)) /* Via pcp? */ @@ -1460,6 +1467,11 @@ static __always_inline bool free_pages_prepare(struct page *page, PAGE_SIZE << order); } + if (get_pageblock_unmapped(page)) { + unmapped_pages_free(page, order); + return false; + } + kernel_poison_pages(page, 1 << order); /* @@ -3525,6 +3537,13 @@ void free_unref_page_list(struct list_head *list) /* Prepare pages for freeing */ list_for_each_entry_safe(page, next, list, lru) { unsigned long pfn = page_to_pfn(page); + + if (get_pageblock_unmapped(page)) { + list_del(&page->lru); + unmapped_pages_free(page, 0); + continue; + } + if (!free_unref_page_prepare(page, pfn, 0)) { list_del(&page->lru); continue; @@ -3856,7 +3875,7 @@ struct page *rmqueue(struct zone *preferred_zone, */ WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1)); - if (likely(pcp_allowed_order(order))) { + if (likely(pcp_allowed(order, gfp_flags))) { /* * MIGRATE_MOVABLE pcplist could have the pages on CMA area and * we need to skip it when CMA area isn't allowed. @@ -5581,6 +5600,11 @@ struct page *__alloc_pages(gfp_t gfp, unsigned int order, int preferred_nid, &alloc_gfp, &alloc_flags)) return NULL; + if (alloc_gfp & __GFP_UNMAPPED) { + page = unmapped_pages_alloc(gfp, order); + goto out; + } + /* * Forbid the first pass from falling back to types that fragment * memory until all local zones are considered. @@ -8437,6 +8461,7 @@ void __init free_area_init(unsigned long *max_zone_pfn) } memmap_init(); + unmapped_alloc_init(); } static int __init cmdline_parse_core(char *p, unsigned long *core, diff --git a/mm/unmapped-alloc.c b/mm/unmapped-alloc.c new file mode 100644 index 000000000000..fb2d54204a3c --- /dev/null +++ b/mm/unmapped-alloc.c @@ -0,0 +1,215 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include + +#include + +#include "internal.h" + +struct unmapped_free_area { + struct list_head free_list; + spinlock_t lock; + unsigned long nr_free; + unsigned long nr_cached; +}; + +static struct unmapped_free_area free_area[MAX_ORDER]; + +static inline void add_to_free_list(struct page *page, unsigned int order) +{ + struct unmapped_free_area *area = &free_area[order]; + + list_add(&page->buddy_list, &area->free_list); + area->nr_free++; +} + +static inline void del_page_from_free_list(struct page *page, unsigned int order) +{ + list_del(&page->buddy_list); + __ClearPageUnmapped(page); + set_page_private(page, 0); + free_area[order].nr_free--; +} + +static inline void set_unmapped_order(struct page *page, unsigned int order) +{ + set_page_private(page, order); + __SetPageUnmapped(page); +} + +static inline bool page_is_unmapped_buddy(struct page *page, struct page *buddy, + unsigned int order) +{ + if (!PageUnmapped(buddy)) + return false; + + if (buddy_order(buddy) != order) + return false; + + return true; +} + +static struct page *find_unmapped_buddy_page_pfn(struct page *page, + unsigned long pfn, + unsigned int order, + unsigned long *buddy_pfn) +{ + unsigned long __buddy_pfn = __find_buddy_pfn(pfn, order); + struct page *buddy; + + buddy = page + (__buddy_pfn - pfn); + if (buddy_pfn) + *buddy_pfn = __buddy_pfn; + + if (page_is_unmapped_buddy(page, buddy, order)) + return buddy; + + return NULL; +} + +static inline void __free_one_page(struct page *page, unsigned int order, + bool cache_refill) +{ + unsigned long pfn = page_to_pfn(page); + unsigned long buddy_pfn; + unsigned long combined_pfn; + struct page *buddy; + unsigned long flags; + + spin_lock_irqsave(&free_area->lock, flags); + + if (cache_refill) { + set_pageblock_unmapped(page); + free_area[order].nr_cached++; + } + + while (order < MAX_ORDER - 1) { + buddy = find_unmapped_buddy_page_pfn(page, pfn, order, + &buddy_pfn); + if (!buddy) + break; + + del_page_from_free_list(buddy, order); + combined_pfn = buddy_pfn & pfn; + page = page + (combined_pfn - pfn); + pfn = combined_pfn; + order++; + } + + set_unmapped_order(page, order); + add_to_free_list(page, order); + spin_unlock_irqrestore(&free_area->lock, flags); +} + +static inline void expand(struct page *page, int low, int high) +{ + unsigned long size = 1 << high; + + while (high > low) { + high--; + size >>= 1; + + add_to_free_list(&page[size], high); + set_unmapped_order(&page[size], high); + } +} + +static struct page *__rmqueue_smallest(unsigned int order) +{ + unsigned int current_order; + struct unmapped_free_area *area; + struct page *page = NULL; + unsigned long flags; + + spin_lock_irqsave(&free_area->lock, flags); + + /* Find a page of the appropriate size in the preferred list */ + for (current_order = order; current_order < MAX_ORDER; ++current_order) { + area = &free_area[current_order]; + page = list_first_entry_or_null(&area->free_list, struct page, + lru); + if (!page) + continue; + + del_page_from_free_list(page, current_order); + expand(page, order, current_order); + + break; + } + + spin_unlock_irqrestore(&free_area->lock, flags); + + return page; +} + +/* FIXME: have PMD_ORDER at last available in include/linux */ +#define PMD_ORDER (PMD_SHIFT - PAGE_SHIFT) + +struct page *unmapped_pages_alloc(gfp_t gfp, int order) +{ + + int cache_order = PMD_ORDER; + struct page *page; + + page = __rmqueue_smallest(order); + if (page) + goto out; + + gfp &= ~__GFP_UNMAPPED; + while (cache_order >= order) { + page = alloc_pages(gfp | __GFP_ZERO, cache_order); + if (page) + break; + cache_order--; + } + + if (page) { + unsigned long addr = (unsigned long)page_address(page); + unsigned long nr_pages = (1 << order); + unsigned long size = PAGE_SIZE * nr_pages; + + /* FIXME: set_direct_map_invalid_noflush may fail */ + for (int i = 0; i < nr_pages; i++) + set_direct_map_invalid_noflush(page + i); + + flush_tlb_kernel_range(addr, addr + size); + + /* + * FIXME: have this under lock so that allocation running + * in parallel won't steal all pages from the newly cached + * ones + */ + __free_one_page(page, cache_order, true); + page = __rmqueue_smallest(order); + + } + +out: + if (page) { + /* FIXME: __prep_new_page() expects page count of 0 */ + set_page_count(page, 0); + post_alloc_hook(page, order, GFP_KERNEL); + } + + return page; +} + +void unmapped_pages_free(struct page *page, int order) +{ + __free_one_page(page, order, false); +} + +int unmapped_alloc_init(void) +{ + for (int order = 0; order < MAX_ORDER; order++) { + struct unmapped_free_area *area = &free_area[order]; + INIT_LIST_HEAD(&area->free_list); + spin_lock_init(&area->lock); + } + + return 0; +} From patchwork Wed Mar 8 09:41:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13165510 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66214C6FD1F for ; Wed, 8 Mar 2023 09:41:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 763836B0078; Wed, 8 Mar 2023 04:41:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6E812280001; Wed, 8 Mar 2023 04:41:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D7756B007D; Wed, 8 Mar 2023 04:41:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4D4006B0078 for ; Wed, 8 Mar 2023 04:41:33 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 10C93A0E9F for ; Wed, 8 Mar 2023 09:41:33 +0000 (UTC) X-FDA: 80545238466.10.CD127C2 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf10.hostedemail.com (Postfix) with ESMTP id 53B84C0010 for ; Wed, 8 Mar 2023 09:41:31 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=a4zMp2Yn; spf=pass (imf10.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678268491; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TCY97IgQdv+eVDd67Oi2qOqfRry5DWncVhqaOuA69eE=; b=Qpw7+Vx2wEZDKpjwUY22T4jM4M1x4wIAhtMqG3DnHYBOZQpm4HPI6KKTLRqkgvaqA/lz6H bgYkr09Aei3DB9abel6zlwis33wiqd1aYc5sMhVO2cVM1cTUDMb1js8CS/Sq94v0is4m1C rw6XGtT2B1WObKaFQyRbCmlZoH6oFrs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=a4zMp2Yn; spf=pass (imf10.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678268491; a=rsa-sha256; cv=none; b=XvtdXoN8VjrB/NAECJ9olvaUd8Wb2jvm7gnXCjJ+QAw4frXxacPNKpq8dnQV3l//NeQOq8 l6PtzoXo5FxbHWeoBax30+eKnvbD67dh/oWxU/5tBwnGR//6gcdXynkArz81DAoAMkybDy XTKYcRURx+kSofxGkABtgG+RMwTW09Y= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1B2CAB81C10; Wed, 8 Mar 2023 09:41:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6E1ACC4339E; Wed, 8 Mar 2023 09:41:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268488; bh=nxbvrZboLg9dZQ5/M7cVi+ZACd7kAmzqj1YWR08+kKk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a4zMp2Yn7HDfIOkHs2LHrjV9yaXYJYlXB/aKnJzLJ/PF5ZrlqK+R5tPO7fbFxNUxs IOg7AHUKv7exmJro4ZoTEYmbvSPOoqa73BwAI14TlGIR/CFSFZ+7ZTm1F0MbW4ELt4 6jMh8OZGtB42IpkD0kOZcf1Woaq6cEJMzUTm34te4vtlC80/UgPKBHW7wvG4rsoFI6 CQmqk2kwLGUSJ4nEjQYF3tw0ODzEx2YVylfv2/ajUcNhSCG99D43hWVU28PhQ7/TUw 8u+3ARAC6O8BItMDilZ/vzwy8gnulVEiyO5ZbkV8lD3YZSazdBAqfut4kR1L6muYwQ DRYHul/m47P0w== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 2/5] mm/unmapped_alloc: add debugfs file similar to /proc/pagetypeinfo Date: Wed, 8 Mar 2023 11:41:03 +0200 Message-Id: <20230308094106.227365-3-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 53B84C0010 X-Stat-Signature: gmrkxscxy9xmup85cfhho6gfd75o15oi X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1678268491-269996 X-HE-Meta: U2FsdGVkX19tQvjYSL/fLoUBZa44I5QV2LAKLIx2ZZg0ic9Cfoo9gLxSAoYhrjxh8CiLxvOFAg1ZwfdnUtAIS8LoncFHOtH8Dl+6esl8kV9NqHy5T4zq7q9jFyOcXojpW8ETk6bXKvFJtCkLNmpyTcRhPrlfCstDRjddd9b3nly9Jaqo0nqDmAZDdT8jveEuurnS2tGu4Msr6jHqM6ge/JzcEEwW/yHgcjzx+LAzwFVlnrFWrcmyq2EtXlXaMBAaxejj5ybTzHBjokG3LGxZcoFccnUVLnn6Wj3KK+HufayawwkhtKjFX9I8EWke0OGbU40QNvyvdXLU20CIWwMQNFmsYh3ajnVQkT+UMj5yAavXxHjpD64ZJ9ijp0xl2yI0+LV3c2ZWyfDGwFsGba4DoO/Sv5QquPUeG+0/pjoU7025S6xZ6++Sd4jjBYHpf+ku9j5/34krnuGLL0HtUS1qsJ3a5SmjyW/Yay+IHr2gyoyX0mOCGkWnNF0QIa6gHYQ2y/JUwu5GGb/po90ZrCpQR6edOa6B2xtvrcu1oLABDMQiEtJC3rIUgVEdXhwvgNwAzWy7kV0Lsar4CnJZEzo69Mv7gjm4kgZx8jM4OaUFk9qe+0UG5teccxZUh/TSO0HuU6TkwS6t5rzkssDox3e3O3Vx8gBaySTuox4gIvQG76pBjtb6YgKBeMjqK3mKqSHZfOmwYa6xolOy8yelzNpr5iAx+6Djm++S7vJnER1roSklirSsoI+Rm+AFqSEfmzUlVWAqy5nrMofqAcHkfhZx70DNlL3lVjJr8Bqs0YSNLjr0XBSzmKBy/aDlTpdH1//M2rPvi0q+2GpuXPl7Ri+YAqiJzN+bBZdTZjn9DHIcWO15HuQJR+t1uVo9ppSvQBYXn+b3yMrt2LMPc+tuHRJ96wCDmEVbRr1bbY6rzT+mcG8obejCui/j0atPksxsT1VSdlHtsURTbPVWDs3jxH9 8A6lgokv LRpKhXurYYphIbzaODlOTNrUWElHo9WuCkSnkdeUQcCqrYxnUINSnR5FMWWXgg5CcS3AwfSoFn7eyZP7l/v1CcsAOhd1KXuCgmVSy6dFegRcZv0idFjelgo5LhVwho31k73U+fa+6ZpUlhRf/SoNluPIt+W1YHMxDP4ttHkgzo7YqgMuyaowjgYelgqPmlepIHsri8eKHriymZV++X5TITR6JevawjSKLujwfWHrGB+S7Ep+BerDqPHub4qOtqKneXbWW8s7kk/yr6rtYD7ghNm/coyqlU9HpT2Cz1Iv3SycLLFKHLWyTHp8OUCLFNgWU8Crg1V+EogGiV77zHPoskgWbXi1XgEAg/ZCpgvJdiTYv+0wKWaejVY5/gSlBcJa0t1ciPmpEXdGaa7gJ+hlR4drk0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Present statistics about unmapped_alloc in debugfs Signed-off-by: Mike Rapoport (IBM) --- mm/unmapped-alloc.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/mm/unmapped-alloc.c b/mm/unmapped-alloc.c index fb2d54204a3c..f74640e9ce9f 100644 --- a/mm/unmapped-alloc.c +++ b/mm/unmapped-alloc.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -213,3 +214,37 @@ int unmapped_alloc_init(void) return 0; } + +static int unmapped_alloc_debug_show(struct seq_file *m, void *private) +{ + int order; + + seq_printf(m, "MAX_ORDER: %d\n", MAX_ORDER); + seq_putc(m, '\n'); + + seq_printf(m, "%-10s", "Order:"); + for (order = 0; order < MAX_ORDER; ++order) + seq_printf(m, "%5d ", order); + seq_putc(m, '\n'); + + seq_printf(m, "%-10s", "Free:"); + for (order = 0; order < MAX_ORDER; ++order) + seq_printf(m, "%5lu ", free_area[order].nr_free); + seq_putc(m, '\n'); + + seq_printf(m, "%-10s", "Cached:"); + for (order = 0; order < MAX_ORDER; ++order) + seq_printf(m, "%5lu ", free_area[order].nr_cached); + seq_putc(m, '\n'); + + return 0; +} +DEFINE_SHOW_ATTRIBUTE(unmapped_alloc_debug); + +static int __init unmapped_alloc_init_late(void) +{ + debugfs_create_file("unmapped_alloc", 0444, NULL, + NULL, &unmapped_alloc_debug_fops); + return 0; +} +late_initcall(unmapped_alloc_init_late); From patchwork Wed Mar 8 09:41:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13165512 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A7D5C64EC4 for ; Wed, 8 Mar 2023 09:41:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A94E16B007B; Wed, 8 Mar 2023 04:41:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A43026B007D; Wed, 8 Mar 2023 04:41:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90C4B6B007E; Wed, 8 Mar 2023 04:41:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 83C726B007B for ; Wed, 8 Mar 2023 04:41:37 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 57088160D82 for ; Wed, 8 Mar 2023 09:41:37 +0000 (UTC) X-FDA: 80545238634.09.3B88A34 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf20.hostedemail.com (Postfix) with ESMTP id 7F49B1C0018 for ; Wed, 8 Mar 2023 09:41:35 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=V6FAK2+t; spf=pass (imf20.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678268495; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h8mhSnmMoDufRPSIQ3CwA06wj1H+ncDCjkrSqDXN2qo=; b=c3buwB3bw30Nvluxd15s/Q3xfBQwdUvju8M+xBaQQfZWyyb6tlSKpushMdIcH6le4iZg0Q OTGnxtbkoghMM7jKwFGr7NqO7a3Vo80/g0NlN+n5Di1btx78wV4lEgrZ2etL9QVAZY7BYv 1aQV8T2qS+m3abeJrMFxtAk9ECtthaY= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=V6FAK2+t; spf=pass (imf20.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678268495; a=rsa-sha256; cv=none; b=cGDOcKwmraKn5cDbizpzjgFZHsCaJDv6HW9AhrwhfyPTcGkjjxiYBrzHfB26GWYCYD4Dcz /9xJ3FzlJ58UiePwTIabH4vTOYhpJ0RbZRwofMnxq0d3jyPHLvLrjbSn594kpl9ezbLonh VovirQ+4J0OEoiCZPWAJXebZlDXmtXM= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 2D5B2B81BFF; Wed, 8 Mar 2023 09:41:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 78284C433A4; Wed, 8 Mar 2023 09:41:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268492; bh=8DLovPxnQy1OCrJZdJAEGK7t0D40yssgi5Mxz0RgWek=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=V6FAK2+t6gSHB91y9VkD/0zOfXhNUhq+sMc5AX4IQuRsdMaWCpYJCn1tHJeqYnOLY RWQ8R3BcCx3KjA7HQiX4FWRwiXskDus+nGuTU/h7Wao7lKZmWu32p94flaMQoKhkjv nL3jFqNz9nbN+CXPwNbIimjP38J1yfRIKOOiHu1l6hqqXu2wHBe6fikO0N0rwDw4rV jpSOfwYDrUpzaXm9lANUnmHiYWQ34T2LFwbVrf8z2Ilhgv3JEnwwcpcBlhfjLrjayl W8CcmajJxQHjkRZKa9pmK4E32ysqR4p5Fy+FD17Vl8xGoLMRnRfcR+TZrgX9jnjemr Axe7yl8DldgkA== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 3/5] mm/unmapped_alloc: add shrinker Date: Wed, 8 Mar 2023 11:41:04 +0200 Message-Id: <20230308094106.227365-4-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: kcu4m5n8k9nm4rf748so3hob4axihyxy X-Rspamd-Queue-Id: 7F49B1C0018 X-HE-Tag: 1678268495-629604 X-HE-Meta: U2FsdGVkX19A3+AzDBdBH9mZwHkBdyz1Qyo5pD86OkiPYqVAZ3f3bVt8NgHIN/fRdRUrkN4crBCq3s+oDA8pxt04cE3SmDCVph9Ixnvc4NdpAPq4xJ3po113m6hbYz/1U+Y0gzNqEDGlPFxmnUvH5HVueIDIt4xkvtvr4kAY0ZiH5K+PWzZXzqmANpCk8nPMEEDcNbaj5Q89P7g2wf3RfYWkLC9bAUvO/0hJO/Hveqar22NM27oA49YBmzvbe26VPihLoF0AVSeUmiBArPOYa/3v8LWikXkyPsa4JLWMvnbpsXeAELB+7anHfPTCrX3w73TLsIjlWOXy/RlZupn47S7tWDtlrMlV4+kIeeGh8Y380rVUeMc0IA65H06yaGSA8WLMxoAl/zJxySgzW7Ud9G593g0HuWZiFMi/eODIrmiOZu68wgB2xvptC7DX2NHCoiLSuA8uL5SmiyxsxzeQrTptthTEAfn4opwydyIFs4FIbHmJMVUrSh4fKka96TwWHOzDTDk/bnwsRVsy/4DXsH8n/vQK8IEVl+Ql6dvX4SOiaOkEtZNpTjYms5NEMhfebZkPJA8tP0xjFdbEVKj8xQqFDgoRDN3mAP0V1u2MPj2LJm6kjHQ0dPJcEmtrZUqQS+OkUGqAuFPrsYFfZ4PLK1no040KOAXIRWiXjJWYm1fFU7Iw38NSbxqGyy6L3Ker16rkCj8IkG+wDKEYJViBLEdqRt1nRa5ptIWJDA/oRbBSYVY54pFN/Tivi+Ney76vQ01ycx2Ga8qAa4NvcYhdiipSoF/oBxT0rjqcUIR6qs8JJGQn5ZvSlQYK4icpkBUVP/goPQBYNdrWaHOJKalUUhGbng0pTmNdaX/PJu2U7hkNm8aoBm0lqLxCDjGSFOk9q4Xz2JvPxfSic8b27fkqwB9DG2TPAyGu68DiWedkbgvJ550CUN92XWs4suSTmbCOA1Ywl6ufuU2OU3iqz92 wBdRxOlH lLe9xsKnQlRf6CFE06BxHhkzdQENgm1UNL7V79kr56/jGEyfSl44gbsY3EVHjtqQhgWJUN4kk6nd/3u+2BAx/lO8N8MpOYzUJOP1tKSsnFvm4QK6jCmvp9cOziRanPUXLrfzuR/1qHWpGJ8xtVRMFOP0Uja7CBjw1RNRQ6ywb3N58A/gauSLY3GZjqEU/uOYABS/M18QaSx5AqRQ2UemlBhqzVrFeg8hsCA/nJBWR684MPSUxTca9LUss2yTw7MIVRAdJDWGcru1U3ay9411GNdl2NRKAvpnVhWBCh6bwoVgGfQjO7sEIdCdj0VMiSKg/VHkCq9BgIeu1evmac2yL6N82S2RdfuBjSj3nmgwzClAiZzeauwb66iwW71yHGr6+bDoLPExzOMkNweqS1SiNp3wzig== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Allow shrinking unmapped caches when there is a memory pressure. Signed-off-by: Mike Rapoport(IBM) --- mm/internal.h | 2 ++ mm/page_alloc.c | 12 ++++++- mm/unmapped-alloc.c | 86 ++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 98 insertions(+), 2 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 8d84cceab467..aa2934594dcf 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1064,4 +1064,6 @@ static inline struct page *unmapped_pages_alloc(gfp_t gfp, int order) static inline void unmapped_pages_free(struct page *page, int order) {} #endif +void __free_unmapped_page(struct page *page, unsigned int order); + #endif /* __MM_INTERNAL_H */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 01f18e7529b0..ece213fac27a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -123,6 +123,11 @@ typedef int __bitwise fpi_t; */ #define FPI_SKIP_KASAN_POISON ((__force fpi_t)BIT(2)) +/* + * Free pages from the unmapped cache + */ +#define FPI_UNMAPPED ((__force fpi_t)BIT(3)) + /* prevent >1 _updater_ of zone percpu pageset ->high and ->batch fields */ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) @@ -1467,7 +1472,7 @@ static __always_inline bool free_pages_prepare(struct page *page, PAGE_SIZE << order); } - if (get_pageblock_unmapped(page)) { + if (!(fpi_flags & FPI_UNMAPPED) && get_pageblock_unmapped(page)) { unmapped_pages_free(page, order); return false; } @@ -1636,6 +1641,11 @@ static void free_one_page(struct zone *zone, spin_unlock_irqrestore(&zone->lock, flags); } +void __free_unmapped_page(struct page *page, unsigned int order) +{ + __free_pages_ok(page, order, FPI_UNMAPPED | FPI_TO_TAIL); +} + static void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid) { diff --git a/mm/unmapped-alloc.c b/mm/unmapped-alloc.c index f74640e9ce9f..89f54383df92 100644 --- a/mm/unmapped-alloc.c +++ b/mm/unmapped-alloc.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include @@ -16,6 +17,7 @@ struct unmapped_free_area { spinlock_t lock; unsigned long nr_free; unsigned long nr_cached; + unsigned long nr_released; }; static struct unmapped_free_area free_area[MAX_ORDER]; @@ -204,6 +206,82 @@ void unmapped_pages_free(struct page *page, int order) __free_one_page(page, order, false); } +static unsigned long unmapped_alloc_count_objects(struct shrinker *sh, + struct shrink_control *sc) +{ + unsigned long pages_to_free = 0; + + for (int order = 0; order < MAX_ORDER; order++) + pages_to_free += (free_area[order].nr_free << order); + + return pages_to_free; +} + +static unsigned long scan_free_area(struct shrink_control *sc, int order) +{ + struct unmapped_free_area *area = &free_area[order]; + unsigned long nr_pages = (1 << order); + unsigned long pages_freed = 0; + unsigned long flags; + struct page *page; + + spin_lock_irqsave(&area->lock, flags); + while (pages_freed < sc->nr_to_scan) { + + page = list_first_entry_or_null(&area->free_list, struct page, + lru); + if (!page) + break; + + del_page_from_free_list(page, order); + expand(page, order, order); + + area->nr_released++; + + if (order == pageblock_order) + clear_pageblock_unmapped(page); + + spin_unlock_irqrestore(&area->lock, flags); + + for (int i = 0; i < nr_pages; i++) + set_direct_map_default_noflush(page + i); + + __free_unmapped_page(page, order); + + pages_freed += nr_pages; + + cond_resched(); + spin_lock_irqsave(&area->lock, flags); + } + + spin_unlock_irqrestore(&area->lock, flags); + + return pages_freed; +} + +static unsigned long unmapped_alloc_scan_objects(struct shrinker *shrinker, + struct shrink_control *sc) +{ + sc->nr_scanned = 0; + + for (int order = 0; order < MAX_ORDER; order++) { + sc->nr_scanned += scan_free_area(sc, order); + + if (sc->nr_scanned >= sc->nr_to_scan) + break; + + sc->nr_to_scan -= sc->nr_scanned; + } + + return sc->nr_scanned ? sc->nr_scanned : SHRINK_STOP; +} + +static struct shrinker shrinker = { + .count_objects = unmapped_alloc_count_objects, + .scan_objects = unmapped_alloc_scan_objects, + .seeks = DEFAULT_SEEKS, +}; + int unmapped_alloc_init(void) { for (int order = 0; order < MAX_ORDER; order++) { @@ -237,6 +315,11 @@ static int unmapped_alloc_debug_show(struct seq_file *m, void *private) seq_printf(m, "%5lu ", free_area[order].nr_cached); seq_putc(m, '\n'); + seq_printf(m, "%-10s", "Released:"); + for (order = 0; order < MAX_ORDER; ++order) + seq_printf(m, "%5lu ", free_area[order].nr_released); + seq_putc(m, '\n'); + return 0; } DEFINE_SHOW_ATTRIBUTE(unmapped_alloc_debug); @@ -245,6 +328,7 @@ static int __init unmapped_alloc_init_late(void) { debugfs_create_file("unmapped_alloc", 0444, NULL, NULL, &unmapped_alloc_debug_fops); - return 0; + + return register_shrinker(&shrinker, "mm-unmapped-alloc"); } late_initcall(unmapped_alloc_init_late); From patchwork Wed Mar 8 09:41:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13165513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1B6BC64EC4 for ; Wed, 8 Mar 2023 09:41:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 95AAB6B007D; Wed, 8 Mar 2023 04:41:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 90ABA6B007E; Wed, 8 Mar 2023 04:41:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D5096B0080; Wed, 8 Mar 2023 04:41:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7048E6B007D for ; Wed, 8 Mar 2023 04:41:40 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 457C440EFA for ; Wed, 8 Mar 2023 09:41:40 +0000 (UTC) X-FDA: 80545238760.18.ED34623 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf25.hostedemail.com (Postfix) with ESMTP id 92AD7A000F for ; Wed, 8 Mar 2023 09:41:38 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Pkmvdm5Y; spf=pass (imf25.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678268498; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9oLvwGB5cLXtiQHKzRF/jATRlN7YLETOtgmzRmuPXRE=; b=K9JFOCJThGGtLAHrCuKOoTGO5Yb/b//ExCcfNDyNy2j1iL37hVg5QGGir7B7t/r7JdmMRl +nPcE0aR7ivv0/xdCTa6EradRYvXfuybfZGRJvTPUD4WJKxabPWh52Co9WV4SGtPjBYW3I 1bTxnMzHSu/kuFE6Kjqze55a9QvO3HQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Pkmvdm5Y; spf=pass (imf25.hostedemail.com: domain of rppt@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678268498; a=rsa-sha256; cv=none; b=oKrkIV7bQjdN+qhDOrz5jwOolpl3Wk8M+w03z2K8EgmW9LawCNKuLqf4J6qEHt/uKTmCzR DVmGXMREZ82vD9ez6LEoXBYM9E6RU0pKII5Wbh5NmC2jZCuEJzxPcnH4fVie02yTsPVKTc AvaIERPGlQ4oO6+Hcxv5QhV8iXcrCDo= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 38F78B81BC0; Wed, 8 Mar 2023 09:41:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7AA97C433A0; Wed, 8 Mar 2023 09:41:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268496; bh=GyhXyClHrKgOJtAqFaxPxOscSnZByfdcCSJ4z9LZSNs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Pkmvdm5Ytxnt5h8jxKV6y+9spytNeNwCgY/lva+XKpCfpv2UDmFkQKXBEAPoZcPry ht6/BanBBGO/Ng+UwA4oJEes3D7Do1tSgu0fOyzsrsuFhVAaq8Fsf0yp5RspB2D/Lu lH3YM2i5lENWXxwTFZya+4DVuWQsaPy6gdGfUDbtqepQy68XZdfv9XAzN+HqlVtRDZ ekAyOV3eAfFITQHTYyjo73txVENSz2Zeqo+39fJfmKQc0sbRYnTO1sCtQWFCpyp6Em ZjhRFv88SpQ75C8j8pLGdjLo+iZuPCKs/sCi988FtUl7CBJI2C/MuBNLv8SxEFu5VZ OMu/uN9ykVk5w== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 4/5] EXPERIMENTAL: x86: use __GFP_UNMAPPED for modele_alloc() Date: Wed, 8 Mar 2023 11:41:05 +0200 Message-Id: <20230308094106.227365-5-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 92AD7A000F X-Rspam-User: X-Stat-Signature: 97394pc4oye4qbm79rmh5ku6qpear37d X-HE-Tag: 1678268498-529956 X-HE-Meta: U2FsdGVkX19I7bC2ZE8eGV9P6ba3NRwnUx1WN6dpM2NtlqUy6YKFdLjh+ygp+xAnPcaVoxDrw+ngKUNgteCW5l/SMYb29fln1a2b/zx9UdRTqRwnDucJuBjFvF9yf9g7gDEM3i86x/TX8eNWoQnd63HI/cTM9PvBL4EDuA1loWjnCcqBpKJBoHaLs3Gn9PU7fnF+g+6jaQfd2TPsZuPRXCjPeltwe5R8m6DrHkZEabPcwJC+nxYkNAQZou5UeRA3o/QbZMARvtIMOfElS4VaX+Nu+aqhjspA+p1yRW3pnMHnt86tY8F/hRRK7wOThj/9q8zATqiIHp4HOnyFOVC/ZwOdMyG3MVhOrUdeuI47Bdl3czJl2bW6C0iJrIFzUVjXnW4p4Aa57DbKMAtZOd6fb9rz/LBg7Wtt050qoHwfa2R/sV7+EOnHeb8fq8YWhnFkVlhkvT+XWgw/Gpkwx3C6X8yavev348XA6szOvoEM5DgJJ92WBONEU/eum1tWmIgkhuxDhtgnI1p3zI+g/Tqx7FjMazPEtCkmD/8i557LbBUm9hQcUGhwVMRfA0IKepNt0HAbzq1c/R2GbkHd/SqllSFPCGHahALqv8fsRfcWQIrkcPvG1r/55xLnc9jVcJlrjCSF3dKn9QbZCZY9XTssjvy/3WLUdcm4WE609XFw4IUC5pel0WY0EGvgXLEZFkA2qwekIgFp0bWldYptb93cBlSHu5gSsDFsvq3Sy3K/q1FyuIv917OrhE4cBqTREMjFIPBF3QFcMpm2Z84OZKjr1RbCPU2UanbVMj5gRNlzuUSbrGCLq2AlQhsIbp3t0DXytAvuSli1IKMjbamCv+r8IAPThro2J2oAKoWuzsS9qiZTAdeM6FD/zhvONZZLLgM2mwKHREU3QEz+FKkhbPdtKNZS5PL7poB6KIDZ63hMmT+R7fE5MGFNFeMwtzPRZeOBVVYdFkRn6AGa7FClDah NVUg+Ss5 ABN6CmidLKYvJ03PPD7RZvIzhTAzeQxAV5EZRCPOTwpzoaCNjvUaz/6vqGKFqfLaIXGqb4L1ZAazyoo6Iam9mHjAfV+ktcYPE5S056w0dVsXXQPCSFhzOYK/4STD8D/drlDQBwIc7Xd4KXhSXMqQ4Vo32tZfdqW2ELTWzLOOcUmZeODVlAbhhHpH9sdKHpI2H1tKVl8tzbA8OvBMnya/DBVPLz1USPFMVqzLO2ed3X4uoric7A84UHzytIgKsA/bFkaL/WqiYpOtDmJRfqNreEvl9CMV+jT7Jweu3f+dlubSIDEiEoGscvCqLRzmZtuM9dMLoW2AhwIpmJnXrw8fki+CySzXdDvg+38liyHaNGIToO/i+svg2Bp9yzrVb0MU0FmDW+IKRTr6OLCgsEK0F4jr6Zw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Signed-off-by: Mike Rapoport (IBM) --- arch/x86/kernel/module.c | 2 +- mm/vmalloc.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index 84ad0e61ba6e..845ed70ba5ab 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -67,7 +67,7 @@ static unsigned long int get_module_load_offset(void) void *module_alloc(unsigned long size) { - gfp_t gfp_mask = GFP_KERNEL; + gfp_t gfp_mask = GFP_KERNEL | __GFP_UNMAPPED; void *p; if (PAGE_ALIGN(size) > MODULES_LEN) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ef910bf349e1..84220ec45ec2 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2892,7 +2892,7 @@ vm_area_alloc_pages(gfp_t gfp, int nid, * to fails, fallback to a single page allocator that is * more permissive. */ - if (!order) { + if (!order && !(gfp & __GFP_UNMAPPED)) { gfp_t bulk_gfp = gfp & ~__GFP_NOFAIL; while (nr_allocated < nr_pages) { From patchwork Wed Mar 8 09:41:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Rapoport X-Patchwork-Id: 13165514 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E59FC64EC4 for ; Wed, 8 Mar 2023 09:41:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A7DA6B007E; Wed, 8 Mar 2023 04:41:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 058996B0080; Wed, 8 Mar 2023 04:41:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E898C6B0081; Wed, 8 Mar 2023 04:41:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DBBF06B007E for ; Wed, 8 Mar 2023 04:41:45 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5481880F3F for ; Wed, 8 Mar 2023 09:41:45 +0000 (UTC) X-FDA: 80545238970.16.76FF078 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf05.hostedemail.com (Postfix) with ESMTP id A9A8810000A for ; Wed, 8 Mar 2023 09:41:43 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=leVKbWeo; spf=pass (imf05.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1678268503; a=rsa-sha256; cv=none; b=wtEFty8AIsBsSRgM7073GVdJTxdMh1h7lepbMY+5eRkSyUNLLWU/OvlcynGri1Psd91T7L Gqkg6eGZkS4Aw2Y1ppAT/YG1lXGPKeaQlbHmAYRjAnV7YClNSzqSB9BKWxkwticbMpoKoN 4R8zQ1inAQ/WCRXeLCG2d3FZxgZOUUA= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=leVKbWeo; spf=pass (imf05.hostedemail.com: domain of rppt@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=rppt@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1678268503; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gMbdMo/pBqYIxP3tjUm5UhHWCZe8JSSf6Iu/9RrXzi8=; b=wdM93gXwK2m37nkQvfOnaqurSs2ZuOJ/6lENV0TDkNZ0+MgIDbwOB+YsbjnZWwB5arqEyf lh9rsdqK8q5sFRSd6aQsS8A4fQ0bVsHHMOZBWJT41R7dgdt76CTkvmraSHrOfJ6A60hcv+ 8H4sZtDuVBVLQJlqoyBQQnBMURkXA4I= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 955DE61721; Wed, 8 Mar 2023 09:41:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80E84C433D2; Wed, 8 Mar 2023 09:41:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1678268500; bh=FOZyjqFesuEnEwpUW1WYq+F0fxn2qMQwF+XDs3hZBQk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=leVKbWeo/7qOftIstG5B9Wgf9MRXnwDGvaQUihQL6sWuanLYUQozOUwMx4mEYxlep SK+6GKBux58RzySSLw+ASPQ2LI0d6QdcBWHG4l2uuE9FqQyRWANesgRaYuCvprRtL2 994aEBWZuA/D8wD0+MLrSqO/8JSauc/E8A48GBC4UFCF0Mteg/IOFl9IH+v5CGnkmX 78RxsHHsDeqbtilOWnIuJihopZEwO+M4DIWXi6/YmWDg7iUB7fVn9xfYNiaJ5WWZ4p tvTaCpCyYZdmV2vKLa1pPMyu6s3qL5dgi4J4Ywy5j0orqUR0qA92pNM5IFhFjn/keq 6l9vE6tUdD9qw== From: Mike Rapoport To: linux-mm@kvack.org Cc: Andrew Morton , Dave Hansen , Mike Rapoport , Peter Zijlstra , Rick Edgecombe , Song Liu , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Subject: [RFC PATCH 5/5] EXPERIMENTAL: mm/secretmem: use __GFP_UNMAPPED Date: Wed, 8 Mar 2023 11:41:06 +0200 Message-Id: <20230308094106.227365-6-rppt@kernel.org> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230308094106.227365-1-rppt@kernel.org> References: <20230308094106.227365-1-rppt@kernel.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: A9A8810000A X-Rspamd-Server: rspam01 X-Stat-Signature: rh84i5cjtsjdde6iwot9b1mpas3u7x9b X-HE-Tag: 1678268503-979268 X-HE-Meta: U2FsdGVkX18L6rLkHulcp/Y0MeZJNil2Yp38vfXvAI7P/QFx92Pe6AS5V6IfbJN5NmC9izO56whXRKzQcDd/gZvUen3uY6rOK5ksfZC4dI0yOdU4MsZRlysHgRKB2Hdyyn8veq0eMvfCB8PsxU7ZuZ5mp7J4ZTe7vK0/30FbQF/12SPwP/rZxdRQ19ZFGniVeurKDwXVdvQmHbcFWex+1f9Mlnntp7G8DxZ1f+JGmDV2rf8Ixjqd4HAM6tWh3t7p8pZ0S2mOJDH+xJFaHlvHiPGvpRaz4heHQnFHWtgSXBs7Gf9geCIXy/XMfIC9Htwe/l4gsJfEnRLJ9N54UgmD+ph62BmrjHA97PtiKGnwxA2S8l0ar72Uo6+xANVW5vugVr2Fn70xL9qZ+MGs68vmRSsv5CpORvo4F2GR9ebP+jSi6B8Qo9WTeOuwxzLlxiJV2aCuzep6Gabs2HVT50scOMSmH5my09uyL6nbU+wRwuJbklI2Gf64t22R5nTfhQd4cl/4Q+k1ihU39NHcWXb5TBNhafWUUPWIC6tZP7xWBZQknwfsphSvMXhOGr/LL46LkhUFZb6NpJROHwhxVCgNbWgIOPK4lzl2iPxjTvyKlOlQ9hrmPXPztBuZFcT7oEyDIW8Hb/RMXNtH5dqh+68G7kSA0hiuahm4OAik2rYmRBQzARQgqRiwQCCaweLqGQmdShTxFLgKduFiXp4CvbGiGDjM5Nc32zyjG1SIBHq6rUea6pkOQTAQ+wyFs2xBsHdUNqP7AsH9w5B/hNsLIzhqLxo7mv91BdPDdvx1w8IlV6zDR39BtubTpCoLkqazFuPqjbB9c5kfinkJvr/C24t4wEd4OUkvbiHznTGm0rtJVsADeLQagP6EvEP/kHVOh1qFXXChWD82ZxXssMYyVs7sdAnWLMAZ397NGbvquVSYrDsyBvOU1jcMzPiKJdgelrkgKiz8W99Y7+M50hPNFUR /zQaBdO9 sE/SJK4X5W+4cYyyNSr4wJM4zVgy1AqNGgw7IpWyG+qmOP2J20ZPw/Esj6AtFcg9vhFFyPTzulb13yXLsPhaprACMx7bwqKPWbdx5XL4oUp5iw4BvzlSmk8g9XileZ3gtVil+FiA9VUREdU9h577tkEmN43vLadTWc0YOZp26jTXu4t6Kv/mh7vyjn7GlE3FidEDC0CA3TLsmDRXHm5e0HXAWkAkVwYrBqTXELN6qef8z1rGMsq8f9U4M2YYLHLt9ftpMMnng/jZUaCaDdXdD2dpunzeT+UU4T0OsD+SxAHQ8XsdbFWb42w+VOy+haOhCtCjxZXvojnfCDPR7sWR5dC6Dipy4OkE6ps6IdugVcltrYDYGKS2fXm53oqHmBDRfKVMeVtLX7iwFPdi5HsH1YayQpg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Mike Rapoport (IBM)" Signed-off-by: Mike Rapoport (IBM) --- mm/secretmem.c | 26 +------------------------- 1 file changed, 1 insertion(+), 25 deletions(-) diff --git a/mm/secretmem.c b/mm/secretmem.c index 0b502625cd30..f66dfd16a0c3 100644 --- a/mm/secretmem.c +++ b/mm/secretmem.c @@ -53,7 +53,6 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) struct inode *inode = file_inode(vmf->vma->vm_file); pgoff_t offset = vmf->pgoff; gfp_t gfp = vmf->gfp_mask; - unsigned long addr; struct page *page; vm_fault_t ret; int err; @@ -66,38 +65,22 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) retry: page = find_lock_page(mapping, offset); if (!page) { - page = alloc_page(gfp | __GFP_ZERO); + page = alloc_page(gfp | __GFP_ZERO | __GFP_UNMAPPED); if (!page) { ret = VM_FAULT_OOM; goto out; } - err = set_direct_map_invalid_noflush(page); - if (err) { - put_page(page); - ret = vmf_error(err); - goto out; - } - __SetPageUptodate(page); err = add_to_page_cache_lru(page, mapping, offset, gfp); if (unlikely(err)) { put_page(page); - /* - * If a split of large page was required, it - * already happened when we marked the page invalid - * which guarantees that this call won't fail - */ - set_direct_map_default_noflush(page); if (err == -EEXIST) goto retry; ret = vmf_error(err); goto out; } - - addr = (unsigned long)page_address(page); - flush_tlb_kernel_range(addr, addr + PAGE_SIZE); } vmf->page = page; @@ -150,15 +133,8 @@ static int secretmem_migrate_folio(struct address_space *mapping, return -EBUSY; } -static void secretmem_free_folio(struct folio *folio) -{ - set_direct_map_default_noflush(&folio->page); - folio_zero_segment(folio, 0, folio_size(folio)); -} - const struct address_space_operations secretmem_aops = { .dirty_folio = noop_dirty_folio, - .free_folio = secretmem_free_folio, .migrate_folio = secretmem_migrate_folio, };