From patchwork Thu Mar 14 14:47:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 13592502 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53E9AC54E67 for ; Thu, 14 Mar 2024 14:46:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A63AE800A9; Thu, 14 Mar 2024 10:46:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A12F26B007B; Thu, 14 Mar 2024 10:46:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86530800A9; Thu, 14 Mar 2024 10:46:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 76D1D6B0071 for ; Thu, 14 Mar 2024 10:46:35 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3ACBE4024C for ; Thu, 14 Mar 2024 14:46:35 +0000 (UTC) X-FDA: 81895920750.21.178D875 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf21.hostedemail.com (Postfix) with ESMTP id 255A71C0003 for ; Thu, 14 Mar 2024 14:46:32 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=hvlaeK3C; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=mCyeGHiM; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=hvlaeK3C; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=mCyeGHiM; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf21.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710427593; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=w/ycXScRm0VEq6f0SxrNdSAEjpI6zCLOzkIxFpkyyFE=; b=67ZiKqb7oiK5eKXEvF4a9j+bMnt0bFCIE+6KUfsNDIkHaV/yHTuN2JDBANZikDBav1NPak QpfWqy1zjUxiDibr3nlh/Km7tlKHVHP+NIm7hnhNJ+Kjx+xmGM+JhSJX5pTUNUpv/cAO+5 VXOy/lkY0bwiva0cKgflPk8mTPAKADc= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=hvlaeK3C; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=mCyeGHiM; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=hvlaeK3C; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=mCyeGHiM; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf21.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710427593; a=rsa-sha256; cv=none; b=uBcN4bZLoXkF06EN7UPTnBPBwCq4N6XtXse5dl9DMbaVXcoqwaM4Tip0J9ApIMbBdB+kEp vV+SaX63qdTSPDhSRJFKrsd2Ozu05gZ52IWdfyrRSora51ASt+s8HLOYAfr2Z0joZD1JcI GvYHJ1QKOx1FoWQwi6inCz3IUVUH5eA= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 469E321CEA; Thu, 14 Mar 2024 14:46:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1710427591; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=w/ycXScRm0VEq6f0SxrNdSAEjpI6zCLOzkIxFpkyyFE=; b=hvlaeK3Cr6vFSGqH4VmQyscRGbrk7hakpsBrj10jkaiynptTMpBRbZ/5aa+/sROncSURpg b64yzx8MarHULW81NUTgawwTshbo71tof3N8tFUZPd9oTsxfNE3O97Wfj6z0U1d34cVYnS 8yFCRr+oSg+2AeJ12VqP9pFcCKnTj3Y= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1710427591; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=w/ycXScRm0VEq6f0SxrNdSAEjpI6zCLOzkIxFpkyyFE=; b=mCyeGHiMRA9mpVFhGedxilqnfxP3ZNzRiRD3Ar6DNG0nxmB/gbbQhPwYzxXLJQ9WGZKie8 A2nxQVsCdblUZJBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1710427591; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=w/ycXScRm0VEq6f0SxrNdSAEjpI6zCLOzkIxFpkyyFE=; b=hvlaeK3Cr6vFSGqH4VmQyscRGbrk7hakpsBrj10jkaiynptTMpBRbZ/5aa+/sROncSURpg b64yzx8MarHULW81NUTgawwTshbo71tof3N8tFUZPd9oTsxfNE3O97Wfj6z0U1d34cVYnS 8yFCRr+oSg+2AeJ12VqP9pFcCKnTj3Y= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1710427591; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=w/ycXScRm0VEq6f0SxrNdSAEjpI6zCLOzkIxFpkyyFE=; b=mCyeGHiMRA9mpVFhGedxilqnfxP3ZNzRiRD3Ar6DNG0nxmB/gbbQhPwYzxXLJQ9WGZKie8 A2nxQVsCdblUZJBw== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id BD3B91368C; Thu, 14 Mar 2024 14:46:30 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 8BCEK8YN82WIHAAAD6G6ig (envelope-from ); Thu, 14 Mar 2024 14:46:30 +0000 From: Oscar Salvador To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Michal Hocko , Vlastimil Babka , Marco Elver , Andrey Konovalov , Alexander Potapenko , Oscar Salvador Subject: [PATCH] mm,page_owner: Fix refcount imbalance Date: Thu, 14 Mar 2024 15:47:53 +0100 Message-ID: <20240314144753.16276-1-osalvador@suse.de> X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-Spamd-Bar: / X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 255A71C0003 X-Stat-Signature: krbo6148gbx4mxc46e78hd59yoeg9ee7 X-HE-Tag: 1710427592-768995 X-HE-Meta: U2FsdGVkX18LD63zfjQKZPz9cLpPCdil6e0n6k5d7sALBThx6j6QdeOKw179uU9JMrfK45kGdYT/5RrEF/WYsrBM4m+f/wLRrjXABbNWl/QROM2eO6vjVLVJkahKhI6ImzcLtHqCDvsNOgh1DApDbVPFY0lw5lXzykM7qmHeg4oqrhLe2zU0uMtPlEB00CSI9Fkko6Ki5FOgXYn98V870VLX07+VoB3KSluT53s+QdKtA1j7Q5T0UJ9GYeUKotEC5q3EzVd5AkSoL8UNt8WcyUH15eTsA2/DUe6bnTCxM9GgsCoIXQlb4Dv0NFdCnY8KxOPbTc+yA7VfLCvvRdtOX9LkQAhGIyviweyXPLuDgqesq8T/HHe5dBKEXPCUHJAwXTsxUDh6FdVSUqfk49BpOFngn9PZ2pmKpNU592o1fmStFBRz2v8iOL9sGOCBPng31SZbfN3PUFz1ulx+w6/DffZXjXqz392Mr7L4zlWA5UHkmz8IMguClEW245qKYEhycSv8IZA9+W0zu4ayFWQQNByw/se0U6J+XI4oevLgYEO18wBmIVdRG8kWPV/SqvVR9dzRaZcdFOxDVD6JszPuOVrgMVJF28Kq5VM2kJwGK1uo/ELoEI9/rt6xZJy7OUy8g6TbdiJ08fVf9GLX67Gx9YGOWfLeEc7IoJFgAI/XnerrBfB16q5RWcgrFaufm6awblosi1LD90IxxR1quItn2Zm+iHkGx6tKsxdV1Esi4B+ISB8UVdmtDwWYqHncWI90F1K3rNKkZdONRVcUYF7au0iu+KNNFFU4QMq3Dkh/+k5mRmXi98poMbqMeP9KoSFuaiByTKJQD09C8GlEx7xBUZhCWyxHnXdZXmjajJTi5GtGNpx3QVevIwdpKJj/AxsljZWrYCS3/EeyK1knYVNQR36wm+hx+1yIHLNYBrJroNKHc9MO9BrYxYPZDgM4zKAtumMsYKjNUoAGAOybcpE pquxNfT7 tXwl7KpbqzzNZMsPzoLJGeBgX/FPZgyIOb5HB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current code does not contemplate scenarios were an allocation and free operation on the same pages do not handle it in the same amount at once. To give an example, page_alloc_exact(), where we will allocate a page of enough order to stafisfy the size request, but we will free the remainings right away. In the above example, we will increment the stack_record refcount only once, but we will decrease it the same number of times as number of unused pages we have to free. This will lead to a warning because of refcount imbalance. Fix this by recording the number of base pages every stack_record holds, and only let the last decrementing of refcount succeed if the number of base pages equals 0, which means we freed all the pages. As a bonus, show the aggregate of stack_count + base_pages as this gives a much better picture of the memory usage. Signed-off-by: Oscar Salvador Fixes: 217b2119b9e2 ("mm,page_owner: implement the tracking of the stacks count") --- include/linux/stackdepot.h | 3 ++ mm/page_owner.c | 57 +++++++++++++++++++++++++++++++------- 2 files changed, 50 insertions(+), 10 deletions(-) diff --git a/include/linux/stackdepot.h b/include/linux/stackdepot.h index 3c6caa5abc7c..261472807c32 100644 --- a/include/linux/stackdepot.h +++ b/include/linux/stackdepot.h @@ -57,6 +57,9 @@ struct stack_record { u32 size; /* Number of stored frames */ union handle_parts handle; /* Constant after initialization */ refcount_t count; +#ifdef CONFIG_PAGE_OWNER + unsigned long nr_base_pages; +#endif union { unsigned long entries[CONFIG_STACKDEPOT_MAX_FRAMES]; /* Frames */ struct { diff --git a/mm/page_owner.c b/mm/page_owner.c index 50111078ecd9..5192449bb581 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -123,10 +123,14 @@ static __init void init_page_owner(void) /* Initialize dummy and failure stacks and link them to stack_list */ dummy_stack.stack_record = __stack_depot_get_stack_record(dummy_handle); failure_stack.stack_record = __stack_depot_get_stack_record(failure_handle); - if (dummy_stack.stack_record) + if (dummy_stack.stack_record) { + dummy_stack.stack_record->nr_base_pages = 0; refcount_set(&dummy_stack.stack_record->count, 1); - if (failure_stack.stack_record) + } + if (failure_stack.stack_record) { + failure_stack.stack_record->nr_base_pages = 0; refcount_set(&failure_stack.stack_record->count, 1); + } dummy_stack.next = &failure_stack; stack_list = &dummy_stack; } @@ -192,9 +196,11 @@ static void add_stack_record_to_list(struct stack_record *stack_record, spin_unlock_irqrestore(&stack_list_lock, flags); } -static void inc_stack_record_count(depot_stack_handle_t handle, gfp_t gfp_mask) +static void inc_stack_record_count(depot_stack_handle_t handle, gfp_t gfp_mask, + unsigned long nr_base_pages) { struct stack_record *stack_record = __stack_depot_get_stack_record(handle); + unsigned long curr_nr_pages; if (!stack_record) return; @@ -209,19 +215,47 @@ static void inc_stack_record_count(depot_stack_handle_t handle, gfp_t gfp_mask) if (refcount_read(&stack_record->count) == REFCOUNT_SATURATED) { int old = REFCOUNT_SATURATED; - if (atomic_try_cmpxchg_relaxed(&stack_record->count.refs, &old, 1)) + if (atomic_try_cmpxchg_relaxed(&stack_record->count.refs, &old, 1)) { /* Add the new stack_record to our list */ add_stack_record_to_list(stack_record, gfp_mask); + smp_store_release(&stack_record->nr_base_pages, + nr_base_pages); + goto inc; + } } + + curr_nr_pages = smp_load_acquire(&stack_record->nr_base_pages); + smp_store_release(&stack_record->nr_base_pages, + curr_nr_pages + nr_base_pages); +inc: refcount_inc(&stack_record->count); } -static void dec_stack_record_count(depot_stack_handle_t handle) +static void dec_stack_record_count(depot_stack_handle_t handle, + unsigned long nr_base_pages) { struct stack_record *stack_record = __stack_depot_get_stack_record(handle); + unsigned long curr_nr_pages; + + if (!stack_record) + return; + + curr_nr_pages = smp_load_acquire(&stack_record->nr_base_pages); + smp_store_release(&stack_record->nr_base_pages, + curr_nr_pages - nr_base_pages); + curr_nr_pages = smp_load_acquire(&stack_record->nr_base_pages); + + /* + * If this stack_record is going to reach a refcount == 1, which means + * free, only do it if all the base pages it allocated were freed. + * E.g: scenarios like THP splitting, or alloc_pages_exact() can have + * an alloc/free operation with different amount of pages + */ + if (refcount_read(&stack_record->count) == 2 && + curr_nr_pages) + return; - if (stack_record) - refcount_dec(&stack_record->count); + refcount_dec(&stack_record->count); } void __reset_page_owner(struct page *page, unsigned short order) @@ -260,7 +294,7 @@ void __reset_page_owner(struct page *page, unsigned short order) * the machinery is not ready yet, we cannot decrement * their refcount either. */ - dec_stack_record_count(alloc_handle); + dec_stack_record_count(alloc_handle, 1UL << order); } static inline void __set_page_owner_handle(struct page_ext *page_ext, @@ -303,7 +337,7 @@ noinline void __set_page_owner(struct page *page, unsigned short order, __set_page_owner_handle(page_ext, handle, order, gfp_mask); page_ext_put(page_ext); set_current_in_page_owner(); - inc_stack_record_count(handle, gfp_mask); + inc_stack_record_count(handle, gfp_mask, 1UL << order); unset_current_in_page_owner(); } @@ -868,6 +902,7 @@ static int stack_print(struct seq_file *m, void *v) struct stack *stack = v; unsigned long *entries; unsigned long nr_entries; + unsigned long nr_base_pages; struct stack_record *stack_record = stack->stack_record; if (!stack->stack_record) @@ -875,6 +910,7 @@ static int stack_print(struct seq_file *m, void *v) nr_entries = stack_record->size; entries = stack_record->entries; + nr_base_pages = stack_record->nr_base_pages; stack_count = refcount_read(&stack_record->count) - 1; if (stack_count < 1 || stack_count < page_owner_stack_threshold) @@ -882,7 +918,8 @@ static int stack_print(struct seq_file *m, void *v) for (i = 0; i < nr_entries; i++) seq_printf(m, " %pS\n", (void *)entries[i]); - seq_printf(m, "stack_count: %d\n\n", stack_count); + seq_printf(m, "stack_count: %d curr_nr_base_pages: %lu\n\n", + stack_count, nr_base_pages); return 0; }