[03/62] mm: Split slab into its own type

Message ID	20211004134650.4031813-4-willy@infradead.org (mailing list archive)
State	New
Headers	show Return-Path: <SRS0=Fowb=OY=kvack.org=owner-linux-mm@kernel.org> DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 598C16121F From: "Matthew Wilcox (Oracle)" <willy@infradead.org> To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" <willy@infradead.org> Subject: [PATCH 03/62] mm: Split slab into its own type Date: Mon, 4 Oct 2021 14:45:51 +0100 Message-Id: <20211004134650.4031813-4-willy@infradead.org> In-Reply-To: <20211004134650.4031813-1-willy@infradead.org> References: <20211004134650.4031813-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: owner-linux-mm@kvack.org Precedence: bulk
Series	Separate struct slab from struct page \| expand [00/62] Separate struct slab from struct page [01/62] mm: Convert page_to_section() to pgflags_section() [02/62] mm: Add pgflags_nid() [03/62] mm: Split slab into its own type [04/62] mm: Add account_slab() and unaccount_slab() [05/62] mm: Convert virt_to_cache() to use struct slab [06/62] mm: Convert __ksize() to struct slab [07/62] mm: Use struct slab in kmem_obj_info() [08/62] mm: Convert check_heap_object() to use struct slab [09/62] mm/slub: Convert process_slab() to take a struct slab [10/62] mm/slub: Convert detached_freelist to use a struct slab [11/62] mm/slub: Convert kfree() to use a struct slab [12/62] mm/slub: Convert __slab_free() to take a struct slab [13/62] mm/slub: Convert new_slab() to return a struct slab [14/62] mm/slub: Convert early_kmem_cache_node_alloc() to use struct slab [15/62] mm/slub: Convert kmem_cache_cpu to struct slab [16/62] mm/slub: Convert show_slab_objects() to struct slab [17/62] mm/slub: Convert validate_slab() to take a struct slab [18/62] mm/slub: Convert count_partial() to struct slab [19/62] mm/slub: Convert bootstrap() to struct slab [20/62] mm/slub: Convert __kmem_cache_do_shrink() to struct slab [21/62] mm/slub: Convert free_partial() to use struct slab [22/62] mm/slub: Convert list_slab_objects() to take a struct slab [23/62] mm/slub: Convert slab_alloc_node() to use a struct slab [24/62] mm/slub: Convert get_freelist() to take a struct slab [25/62] mm/slub: Convert node_match() to take a struct slab [26/62] mm/slub: Convert slab flushing to struct slab [27/62] mm/slub: Convert __unfreeze_partials to take a struct slab [28/62] mm/slub: Convert deactivate_slab() to take a struct slab [29/62] mm/slub: Convert acquire_slab() to take a struct page [30/62] mm/slub: Convert partial slab management to struct slab [31/62] mm/slub: Convert slab freeing to struct slab [32/62] mm/slub: Convert shuffle_freelist to struct slab [33/62] mm/slub: Remove struct page argument to next_freelist_entry() [34/62] mm/slub: Remove struct page argument from setup_object() [35/62] mm/slub: Convert freelist_corrupted() to struct slab [36/62] mm/slub: Convert full slab management to struct slab [37/62] mm/slub: Convert free_consistency_checks() to take a struct slab [38/62] mm/slub: Convert alloc_debug_processing() to struct slab [39/62] mm/slub: Convert check_object() to struct slab [40/62] mm/slub: Convert on_freelist() to struct slab [41/62] mm/slub: Convert check_slab() to struct slab [42/62] mm/slub: Convert check_valid_pointer() to struct slab [43/62] mm/slub: Convert object_err() to take a struct slab [44/62] mm/slub: Convert print_trailer() to struct slab [45/62] mm/slub: Convert slab_err() to take a struct slab [46/62] mm/slub: Convert print_page_info() to print_slab_info() [47/62] mm/slub: Convert trace() to take a struct slab [48/62] mm/slub: Convert cmpxchg_double_slab to struct slab [49/62] mm/slub: Convert get_map() and __fill_map() to struct slab [50/62] mm/slub: Convert slab_lock() and slab_unlock() to struct slab [51/62] mm/slub: Convert setup_page_debug() to setup_slab_debug() [52/62] mm/slub: Convert pfmemalloc_match() to take a struct slab [53/62] mm/slub: Remove pfmemalloc_match_unsafe() [54/62] mm: Convert slab to use struct slab [55/62] mm: Convert slob to use struct slab [56/62] mm: Convert slub to use struct slab [57/62] memcg: Convert object cgroups from struct page to struct slab [58/62] mm/kasan: Convert to struct slab [59/62] zsmalloc: Stop using slab fields in struct page [60/62] bootmem: Use page->index instead of page->freelist [61/62] iommu: Use put_pages_list [62/62] mm: Remove slab from struct page

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 7f8ee09c711f..c2ea71aba84e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -239,6 +239,62 @@ struct page { #endif } _struct_page_alignment; +/* Reuses the bits in struct page */ +struct slab { + unsigned long flags; + union { + struct list_head slab_list; + struct { /* Partial pages */ + struct slab *next; +#ifdef CONFIG_64BIT + int slabs; /* Nr of slabs left */ + int pobjects; /* Approximate count */ +#else + short int slabs; + short int pobjects; +#endif + }; + struct rcu_head rcu_head; + }; + struct kmem_cache *slab_cache; /* not slob */ + /* Double-word boundary */ + void *freelist; /* first free object */ + union { + void *s_mem; /* slab: first object */ + unsigned long counters; /* SLUB */ + struct { /* SLUB */ + unsigned inuse:16; + unsigned objects:15; + unsigned frozen:1; + }; + }; + + union { + unsigned int active; /* SLAB */ + int units; /* SLOB */ + }; + atomic_t _refcount; +#ifdef CONFIG_MEMCG + unsigned long memcg_data; +#endif +}; + +#define SLAB_MATCH(pg, sl) \ + static_assert(offsetof(struct page, pg) == offsetof(struct slab, sl)) +SLAB_MATCH(flags, flags); +SLAB_MATCH(compound_head, slab_list); /* Ensure bit 0 is clear */ +SLAB_MATCH(slab_list, slab_list); +SLAB_MATCH(rcu_head, rcu_head); +SLAB_MATCH(slab_cache, slab_cache); +SLAB_MATCH(s_mem, s_mem); +SLAB_MATCH(active, active); +SLAB_MATCH(_refcount, _refcount); +#ifdef CONFIG_MEMCG +SLAB_MATCH(memcg_data, memcg_data); +#endif +#undef SLAB_MATCH +static_assert(sizeof(struct slab) <= sizeof(struct page)); + static inline atomic_t *compound_mapcount_ptr(struct page *page) { return &page[1].compound_mapcount; diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a558d67ee86f..57bdb1eb2f29 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -165,6 +165,8 @@ enum pageflags { /* Remapped by swiotlb-xen. */ PG_xen_remapped = PG_owner_priv_1, + /* SLAB / SLUB / SLOB */ + PG_pfmemalloc = PG_active, /* SLOB */ PG_slob_free = PG_private, @@ -193,6 +195,33 @@ static inline unsigned long _compound_head(const struct page *page) #define compound_head(page) ((typeof(page))_compound_head(page)) +/** + * page_slab - Converts from page to slab. + * @p: The page. + * + * This function cannot be called on a NULL pointer. It can be called + * on a non-slab page; the caller should check is_slab() to be sure + * that the slab really is a slab. + * + * Return: The slab which contains this page. + */ +#define page_slab(p) (_Generic((p), \ + const struct page *: (const struct slab *)_compound_head(p), \ + struct page *: (struct slab *)_compound_head(p))) + +/** + * slab_page - The first struct page allocated for a slab + * @slab: The slab. + * + * Slabs are allocated as one-or-more pages. It is occasionally necessary + * to convert back to a struct page in order to communicate with the rest + * of the mm. Please use this helper function instead of casting yourself, + * as the implementation may change in the future. + */ +#define slab_page(s) (_Generic((s), \ + const struct slab *: (const struct page *)s, \ + struct slab *: (struct page *)s)) + static __always_inline int PageTail(struct page *page) { return READ_ONCE(page->compound_head) & 1; diff --git a/mm/slab.h b/mm/slab.h index 58c01a34e5b8..54b05f4d9eb5 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -5,6 +5,79 @@ * Internal slab definitions */ +/* + * Does this memory belong to a slab cache? Slub can return page allocator + * memory for certain size allocations. + */ +static inline bool slab_test_cache(const struct slab *slab) +{ + return test_bit(PG_slab, &slab->flags); +} + +static inline bool slab_test_multi_page(const struct slab *slab) +{ + return test_bit(PG_head, &slab->flags); +} + +/* + * If network-based swap is enabled, sl*b must keep track of whether pages + * were allocated from pfmemalloc reserves. + */ +static inline bool slab_test_pfmemalloc(const struct slab *slab) +{ + return test_bit(PG_pfmemalloc, &slab->flags); +} + +static inline void slab_set_pfmemalloc(struct slab *slab) +{ + set_bit(PG_pfmemalloc, &slab->flags); +} + +static inline void slab_clear_pfmemalloc(struct slab *slab) +{ + clear_bit(PG_pfmemalloc, &slab->flags); +} + +static inline void __slab_clear_pfmemalloc(struct slab *slab) +{ + __clear_bit(PG_pfmemalloc, &slab->flags); +} + +static inline void *slab_address(const struct slab *slab) +{ + return page_address(slab_page(slab)); +} + +static inline int slab_nid(const struct slab *slab) +{ + return pgflags_nid(slab->flags); +} + +static inline pg_data_t *slab_pgdat(const struct slab *slab) +{ + return NODE_DATA(slab_nid(slab)); +} + +static inline struct slab *virt_to_slab(const void *addr) +{ + struct page *page = virt_to_page(addr); + + return page_slab(page); +} + +static inline int slab_order(const struct slab *slab) +{ + if (!slab_test_multi_page(slab)) + return 0; + return ((struct page *)slab)[1].compound_order; +} + +static inline size_t slab_size(const struct slab *slab) +{ + return PAGE_SIZE << slab_order(slab); +} + + #ifdef CONFIG_SLOB /* * Common fields provided in kmem_cache by all slab allocators diff --git a/mm/slub.c b/mm/slub.c index 3d2025f7163b..7e429a31b326 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3755,7 +3755,7 @@ static unsigned int slub_min_objects; * requested a higher minimum order then we start with that one instead of * the smallest order which will fit the object. */ -static inline unsigned int slab_order(unsigned int size, +static inline unsigned int calc_slab_order(unsigned int size, unsigned int min_objects, unsigned int max_order, unsigned int fract_leftover) { @@ -3819,7 +3819,7 @@ static inline int calculate_order(unsigned int size) fraction = 16; while (fraction >= 4) { - order = slab_order(size, min_objects, + order = calc_slab_order(size, min_objects, slub_max_order, fraction); if (order <= slub_max_order) return order; @@ -3832,14 +3832,14 @@ static inline int calculate_order(unsigned int size) * We were unable to place multiple objects in a slab. Now * lets see if we can place a single object there. */ - order = slab_order(size, 1, slub_max_order, 1); + order = calc_slab_order(size, 1, slub_max_order, 1); if (order <= slub_max_order) return order; /* * Doh this slab cannot be placed using slub_max_order. */ - order = slab_order(size, 1, MAX_ORDER, 1); + order = calc_slab_order(size, 1, MAX_ORDER, 1); if (order < MAX_ORDER) return order; return -ENOSYS;

[03/62] mm: Split slab into its own type

Commit Message

Comments

Patch