From patchwork Tue Mar 26 06:30:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 13603616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF64FC54E58 for ; Tue, 26 Mar 2024 06:29:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58A266B0098; Tue, 26 Mar 2024 02:29:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53AE96B0099; Tue, 26 Mar 2024 02:29:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38DAE6B009A; Tue, 26 Mar 2024 02:29:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 246DC6B0098 for ; Tue, 26 Mar 2024 02:29:20 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E2E4816098D for ; Tue, 26 Mar 2024 06:29:19 +0000 (UTC) X-FDA: 81938213238.14.0EFD72C Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf11.hostedemail.com (Postfix) with ESMTP id F3E6540011 for ; Tue, 26 Mar 2024 06:29:17 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf11.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711434558; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=D38A+aEqBI0MgZeDH1n4FjWBib0rpEIOMg8Zn4pGai0=; b=npon8iWdzUcX8jDLnuDZnYh4i1Ej9Hqy2Qk9/NNbzaPPIyRcO0ByzPm7vvsSjrpJBdYs7G taDZnUlqSeAq7YjQe0sNxYuu+oJDW4bcjBv2I/+FLrGfh4daAnYflTWFYD/w7W40PGFglK S57O0s8vzM8lCaDOAIkC2QrMDsRaVbo= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf11.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711434558; a=rsa-sha256; cv=none; b=C7tfKE7w8z5gAPG86EZiEKEsdslzsryV2V/BkJikbQN3bQr1HX+FM2XCV1dtUwAWAjSwiB 4kqLjEX+y9BuvZNi1WL7d82TbGO0UUlnmWQYyFef1DXNL9g1T/9gDJb8/8kUiaVwpWXx4J AgJAm1ucbGM95SbdCLGXnG6EbBsJCeA= Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 8622D5D1CD; Tue, 26 Mar 2024 06:29:16 +0000 (UTC) Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 067E413215; Tue, 26 Mar 2024 06:29:15 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap2.dmz-prg2.suse.org with ESMTPSA id MKx6OjtrAmY2YQAAn2gu4w (envelope-from ); Tue, 26 Mar 2024 06:29:15 +0000 From: Oscar Salvador To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Michal Hocko , Vlastimil Babka , Marco Elver , Andrey Konovalov , Alexander Potapenko , Oscar Salvador Subject: [PATCH v3 1/3] mm,page_owner: Update metada for tail pages Date: Tue, 26 Mar 2024 07:30:34 +0100 Message-ID: <20240326063036.6242-2-osalvador@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240326063036.6242-1-osalvador@suse.de> References: <20240326063036.6242-1-osalvador@suse.de> MIME-Version: 1.0 X-Rspamd-Queue-Id: F3E6540011 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: pef3xk6bmu4m9dqk83usf36gxu8s89x7 X-HE-Tag: 1711434557-716393 X-HE-Meta: U2FsdGVkX19py4+zHr+InlNrqaFg9hMGRywHGRuD3fw1TDAwcCg6SQ3DLFbJK5ZTjKz+EWf7rAeDW5pH9XYu8La7OuNgf1lcYzMdXU/ACSeRONIhIahZQPvRvePe9krQBt44FQBxsRFjnESqEkse92LG8ljs2nBUubbkvdYlVI7DKH/SIcwaHAsEcFgHDyzE37VCfgRjMXmMApIasEz4FWbthDS6/kNhvMnuQXxZcGvc9IGYcpViRy8q+VGBvyjeReChc34oBsXUaEBj84kjdwIFW0LNt3TOzcopa1jPNm/NPc0nNeTcLTRON1XlXeEGm14fqHw+ktDE6bBDEdyGBH4waRw2S/pxElVhV2oKRCRsAI6H0Q/8JOtT/b1CzxGIgoEAEoMqUEP/yXvvHLIqwZCMRL4qMhURIjFNA9zs3bGcKnEbCi4isZknR2s516Eveb6mbMm5KKoou8jeJcZQsRvHoLdZz9hpuBawdgOtnoj5a2PKU1Lfh9+HSGTwbJT2PrL+2xttcj0ZDvmJTg5ssplJhIuIEFAloh2zv/EQi3vf6n3tGhYczvfWXLCFfISN/njqndAM5fQIr3lSXNcyvrsRf6NSaxoyh2eG1jbKDiH5ACUzeCMiY5B3WzOfFqnK1z2vossMhc4e9yo0pVzeOWccvUG18jWnhdvQ49UY1Geif3VBRoMaTXTn1U4FGyHMU1CzExiiOjDpm/ddZ87Kw8j12Fbf0LcMC3aBiCNM6thFV4StYE+YtHDquCGxvd5tXQKz1LhRQmbNG1z7hvn+GWpsR0gxrtEyJwIdN9nt6zCREj5rUuUpqDx9bw7vSzBU0F2GKrOBVk/x52OJOS1l9xgCmINE0pcwRHAAcMF0DEMQbDPvU2xKQdmSaaxm134S3qkWAOmDbMl63XcaHqAptw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: __set_page_owner_handle() and __reset_page_owner() update the metadata of all pages when the page is of a higher-order, but we miss to do the same when the pages are migrated. __folio_copy_owner() only updates the metadata of the head page, meaning that the information stored in the first page and the tail pages will not match. Strictly speaking that is not a big problem because 1) we do not print tail pages and 2) upon splitting all tail pages will inherit the metada of the head page, but it is better to have all metadata in check should there be any problem, so it can ease debugging. For that purpose, a couple of helpers are created __update_page_owner_handle() which updates the metadata on allocation, and __update_page_owner_free_handle() which does the same when the page is freed. __folio_copy_owner() will make use of both as it needs to entirely replace the page_owner metadata for the new page. Signed-off-by: Oscar Salvador Reviewed-by: Vlastimil Babka --- mm/page_owner.c | 137 ++++++++++++++++++++++++++---------------------- 1 file changed, 74 insertions(+), 63 deletions(-) diff --git a/mm/page_owner.c b/mm/page_owner.c index d17d1351ec84..52d1ced0b57f 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -228,9 +228,58 @@ static void dec_stack_record_count(depot_stack_handle_t handle) refcount_dec(&stack_record->count); } -void __reset_page_owner(struct page *page, unsigned short order) +static inline void __update_page_owner_handle(struct page_ext *page_ext, + depot_stack_handle_t handle, + unsigned short order, + gfp_t gfp_mask, + short last_migrate_reason, u64 ts_nsec, + pid_t pid, pid_t tgid, char *comm) { int i; + struct page_owner *page_owner; + + for (i = 0; i < (1 << order); i++) { + page_owner = get_page_owner(page_ext); + page_owner->handle = handle; + page_owner->order = order; + page_owner->gfp_mask = gfp_mask; + page_owner->last_migrate_reason = last_migrate_reason; + page_owner->pid = pid; + page_owner->tgid = tgid; + page_owner->ts_nsec = ts_nsec; + strscpy(page_owner->comm, comm, + sizeof(page_owner->comm)); + __set_bit(PAGE_EXT_OWNER, &page_ext->flags); + __set_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); + page_ext = page_ext_next(page_ext); + } +} + +static inline void __update_page_owner_free_handle(struct page_ext *page_ext, + depot_stack_handle_t handle, + unsigned short order, + pid_t pid, pid_t tgid, + u64 free_ts_nsec) +{ + int i; + struct page_owner *page_owner; + + for (i = 0; i < (1 << order); i++) { + page_owner = get_page_owner(page_ext); + /* Only __reset_page_owner() wants to clear the bit */ + if (handle) { + __clear_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); + page_owner->free_handle = handle; + } + page_owner->free_ts_nsec = free_ts_nsec; + page_owner->free_pid = current->pid; + page_owner->free_tgid = current->tgid; + page_ext = page_ext_next(page_ext); + } +} + +void __reset_page_owner(struct page *page, unsigned short order) +{ struct page_ext *page_ext; depot_stack_handle_t handle; depot_stack_handle_t alloc_handle; @@ -245,16 +294,10 @@ void __reset_page_owner(struct page *page, unsigned short order) alloc_handle = page_owner->handle; handle = save_stack(GFP_NOWAIT | __GFP_NOWARN); - for (i = 0; i < (1 << order); i++) { - __clear_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); - page_owner->free_handle = handle; - page_owner->free_ts_nsec = free_ts_nsec; - page_owner->free_pid = current->pid; - page_owner->free_tgid = current->tgid; - page_ext = page_ext_next(page_ext); - page_owner = get_page_owner(page_ext); - } + __update_page_owner_free_handle(page_ext, handle, order, current->pid, + current->tgid, free_ts_nsec); page_ext_put(page_ext); + if (alloc_handle != early_handle) /* * early_handle is being set as a handle for all those @@ -266,36 +309,11 @@ void __reset_page_owner(struct page *page, unsigned short order) dec_stack_record_count(alloc_handle); } -static inline void __set_page_owner_handle(struct page_ext *page_ext, - depot_stack_handle_t handle, - unsigned short order, gfp_t gfp_mask) -{ - struct page_owner *page_owner; - int i; - u64 ts_nsec = local_clock(); - - for (i = 0; i < (1 << order); i++) { - page_owner = get_page_owner(page_ext); - page_owner->handle = handle; - page_owner->order = order; - page_owner->gfp_mask = gfp_mask; - page_owner->last_migrate_reason = -1; - page_owner->pid = current->pid; - page_owner->tgid = current->tgid; - page_owner->ts_nsec = ts_nsec; - strscpy(page_owner->comm, current->comm, - sizeof(page_owner->comm)); - __set_bit(PAGE_EXT_OWNER, &page_ext->flags); - __set_bit(PAGE_EXT_OWNER_ALLOCATED, &page_ext->flags); - - page_ext = page_ext_next(page_ext); - } -} - noinline void __set_page_owner(struct page *page, unsigned short order, gfp_t gfp_mask) { struct page_ext *page_ext; + u64 ts_nsec = local_clock(); depot_stack_handle_t handle; handle = save_stack(gfp_mask); @@ -303,7 +321,9 @@ noinline void __set_page_owner(struct page *page, unsigned short order, page_ext = page_ext_get(page); if (unlikely(!page_ext)) return; - __set_page_owner_handle(page_ext, handle, order, gfp_mask); + __update_page_owner_handle(page_ext, handle, order, gfp_mask, -1, + current->pid, current->tgid, ts_nsec, + current->comm); page_ext_put(page_ext); inc_stack_record_count(handle, gfp_mask); } @@ -342,7 +362,7 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) { struct page_ext *old_ext; struct page_ext *new_ext; - struct page_owner *old_page_owner, *new_page_owner; + struct page_owner *old_page_owner; old_ext = page_ext_get(&old->page); if (unlikely(!old_ext)) @@ -355,31 +375,21 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) } old_page_owner = get_page_owner(old_ext); - new_page_owner = get_page_owner(new_ext); - new_page_owner->order = old_page_owner->order; - new_page_owner->gfp_mask = old_page_owner->gfp_mask; - new_page_owner->last_migrate_reason = - old_page_owner->last_migrate_reason; - new_page_owner->handle = old_page_owner->handle; - new_page_owner->pid = old_page_owner->pid; - new_page_owner->tgid = old_page_owner->tgid; - new_page_owner->free_pid = old_page_owner->free_pid; - new_page_owner->free_tgid = old_page_owner->free_tgid; - new_page_owner->ts_nsec = old_page_owner->ts_nsec; - new_page_owner->free_ts_nsec = old_page_owner->ts_nsec; - strcpy(new_page_owner->comm, old_page_owner->comm); - + __update_page_owner_handle(new_ext, old_page_owner->handle, + old_page_owner->order, old_page_owner->gfp_mask, + old_page_owner->last_migrate_reason, + old_page_owner->ts_nsec, old_page_owner->pid, + old_page_owner->tgid, old_page_owner->comm); /* - * We don't clear the bit on the old folio as it's going to be freed - * after migration. Until then, the info can be useful in case of - * a bug, and the overall stats will be off a bit only temporarily. - * Also, migrate_misplaced_transhuge_page() can still fail the - * migration and then we want the old folio to retain the info. But - * in that case we also don't need to explicitly clear the info from - * the new page, which will be freed. + * Do not proactively clear PAGE_EXT_OWNER{_ALLOCATED} bits as the folio + * will be freed after migration. Keep them until then as they may be + * useful. */ - __set_bit(PAGE_EXT_OWNER, &new_ext->flags); - __set_bit(PAGE_EXT_OWNER_ALLOCATED, &new_ext->flags); + __update_page_owner_free_handle(new_ext, 0, old_page_owner->order, + old_page_owner->free_pid, + old_page_owner->free_tgid, + old_page_owner->free_ts_nsec); + page_ext_put(new_ext); page_ext_put(old_ext); } @@ -787,8 +797,9 @@ static void init_pages_in_zone(pg_data_t *pgdat, struct zone *zone) goto ext_put_continue; /* Found early allocated page */ - __set_page_owner_handle(page_ext, early_handle, - 0, 0); + __update_page_owner_handle(page_ext, early_handle, 0, 0, + -1, local_clock(), current->pid, + current->tgid, current->comm); count++; ext_put_continue: page_ext_put(page_ext); From patchwork Tue Mar 26 06:30:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 13603617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F609C54E58 for ; Tue, 26 Mar 2024 06:29:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 00C2C6B0099; Tue, 26 Mar 2024 02:29:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EFDCE6B009A; Tue, 26 Mar 2024 02:29:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D77D66B009B; Tue, 26 Mar 2024 02:29:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C5E216B0099 for ; Tue, 26 Mar 2024 02:29:23 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7D8B116022D for ; Tue, 26 Mar 2024 06:29:23 +0000 (UTC) X-FDA: 81938213406.01.C5F4DFF Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf07.hostedemail.com (Postfix) with ESMTP id 4B32840010 for ; Tue, 26 Mar 2024 06:29:20 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=uys7I63q; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=o4yifOMq; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=uys7I63q; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=o4yifOMq; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf07.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711434560; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V2s5Z+mBTGRB1b77yA719SIjqvuRSMVvpx566llr28g=; b=uADzVZj/7zVeiLo7ug+Sp4/vumsG2b9HdW3ZI/cLniC/OGyoW7mih5XRaaaJMWKVXpK68d 929AqRbACeiSaTA0qNvUSa5mZWk2jiY791qaCftaBYfVx+Dn7hQBofV+s3QApMN++n1v4l va2E4vrBjNR6O7vNzLNba56R+G32HQ8= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=uys7I63q; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=o4yifOMq; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=uys7I63q; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=o4yifOMq; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf07.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.130 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711434560; a=rsa-sha256; cv=none; b=rC2Oq3E24orNmix1U0XjzAh6Ng601TK+bgNtsT+FngbC3CFvhcwlURLBbG9TrKCNL975yb pKCkt9gUeppXrOYiC41nMzNB0BCBhT15T5BPHIVdJclmLG3xEaB6ZbU8BqqA5YyUdpLlQy HBGItVbwcayml1QKPPIKJFrOJ7hnlyU= Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B964F37809; Tue, 26 Mar 2024 06:29:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1711434558; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V2s5Z+mBTGRB1b77yA719SIjqvuRSMVvpx566llr28g=; b=uys7I63qEzH54ztueaqvOcRv7kEMi7JKK98QUJwjFcOQCbCvkJzWOsZdmawTS01a4bLghx HbMaGF0J90jgQy5t8lYEdlgfTxl7RAxhvuS9ZV0sIWyulS3s1DojetzdlH384Ch+i6+aXt myvKJra6Z4vadQJopHu1BBeZfZixHdA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1711434558; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V2s5Z+mBTGRB1b77yA719SIjqvuRSMVvpx566llr28g=; b=o4yifOMqEqPGNbaCq9D0Jv4FpBOK2yvang64LxEzR7ta7Hr7knVIVmCKpEA3kgloOHZYVI 3jLcWOULZ0oq19CA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1711434558; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V2s5Z+mBTGRB1b77yA719SIjqvuRSMVvpx566llr28g=; b=uys7I63qEzH54ztueaqvOcRv7kEMi7JKK98QUJwjFcOQCbCvkJzWOsZdmawTS01a4bLghx HbMaGF0J90jgQy5t8lYEdlgfTxl7RAxhvuS9ZV0sIWyulS3s1DojetzdlH384Ch+i6+aXt myvKJra6Z4vadQJopHu1BBeZfZixHdA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1711434558; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V2s5Z+mBTGRB1b77yA719SIjqvuRSMVvpx566llr28g=; b=o4yifOMqEqPGNbaCq9D0Jv4FpBOK2yvang64LxEzR7ta7Hr7knVIVmCKpEA3kgloOHZYVI 3jLcWOULZ0oq19CA== Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 24BB013215; Tue, 26 Mar 2024 06:29:18 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap2.dmz-prg2.suse.org with ESMTPSA id 6NRLBj5rAmY2YQAAn2gu4w (envelope-from ); Tue, 26 Mar 2024 06:29:18 +0000 From: Oscar Salvador To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Michal Hocko , Vlastimil Babka , Marco Elver , Andrey Konovalov , Alexander Potapenko , Oscar Salvador , syzbot+41bbfdb8d41003d12c0f@syzkaller.appspotmail.com Subject: [PATCH v3 2/3] mm,page_owner: Fix refcount imbalance Date: Tue, 26 Mar 2024 07:30:35 +0100 Message-ID: <20240326063036.6242-3-osalvador@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240326063036.6242-1-osalvador@suse.de> References: <20240326063036.6242-1-osalvador@suse.de> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 4B32840010 X-Stat-Signature: 9qqofhtbbswbsaa1o1yf9b837jd1k51j X-Rspam-User: X-HE-Tag: 1711434560-536858 X-HE-Meta: U2FsdGVkX19TmSWeIuRPxCVaX8z6kaWkJsOGuDaKrv5wIHdYjGRiURolaIeZDR7bJiJ3ITSef7wacEm9AlBfDTEM/V/FcosvLTEV9midxX5gE5Putw+kjdVm/tcPJzzjjfwhmpHpDRNyTcpiz6lN3VqR4pvfDS4G/dR3RBTBVmSegtKF+/rH0TEcvqZJ8fVk2MEjNhwALkxip3585GcWxoQYuNzgmNdLYPHmY2V/CWM9a8m0nEzWMfsRFhgNGq5QRyGG8YgxGPP09ENaFZhhaTEfB7waO9KCd+nwEuRi1ZC9AXBAODARLWSXPGc9GEratyD5IQzYMcj5uSPA/srMEc49SQqVbmhbTeSX5Iv0UAaiWxtSi2KhsSCSy/XbGLRo5rHaxc7yCacwhBzvVrHZtMXroptw19JZU7gPWR5sPxtfoX9AFiJ6ZdVVbtdihKFxFiI4c39n/pPz+lzjsWgpkLZbtLByeXhNv6r+2NBYEbMSBxKr+pLjzTyN6aS1FCo/XWiKt2Dmu96U1jT5RtUVP3MCNBfVp75lAyKuBeJihsLDfmbW1Vy/r5p7oCajPLJqnND6ie49Z3tkmWyCQO5qn4wlfxzRLGFBON+Z8IBbGlrkP7+DPOYet+gqngjsICqmlhtXe7FyHUbFyIkig/VejYFwMQcb69qb0xoZEVULCDKUQiZK0h9rlV4O4BbU2tjq8MynqWJz8zPuOpEjKQzR45RbHS/3p0/Xze5nVZHC6nRwTr+O86bF9ZmN8Ah38KciG0TynAcOkYvF5kx/ZSJK3uIPoeu+tzRZmhiHy8Dy1NSPokkiprY8n5kLetRWId7yzFlEfrBYmphr/EAkOp0A+NbBDx7c5dI6bW7vng7Tt4IAh5ghrQi6MnK/ubkyPt3pZ9NV9NCWXgwePiShsJrPVXqFimkVVh5UM+x60yN+UhXdIYILnnrmQ0GjJdTBk4VTId6kl2+CUrO5l+rSdAp E9oH5kAR vizb7Fmux//dXBUiDCnn6hevSs+fa6EXFEtmnmFj0Tbj9vBzAYMiKprRXb97fOq4+BrpU6wFmk/xIqI9E0RK7ew5ytkxGaHJ2VSGRi4vZCbBvBVZ8vrWggU3jX00C5yv68fiCQaghmZ7P+DUJ/FErKqOQFYMOptTkI5cFRjmNhgrjobIoxdiR5oFVww7/s0txK1ReWFXTC9O7PDzivc5PQzyltG0jyrm9jjZ3SrsNRYT7aP90D2GQLNlRdNrhxXSgwm3NIPzAsHZXeKYkRkIucDHH1VZnV/XZMZ0LKaQjbNf1QPO8++Fm57ufdlM5x1XfQ8od8E6+IAM8crdk8xvol78qFbudkymvpxuLC+FhH2CgMZCCsM85OaczCnI2duiM4rSc3qyqGnAdwLQRsLeIqY8nfNlUqz3Rqtl7AvSpMLvVbHr/2YLjrYfCEPZY9f9maiOiwKua1QhiBHlitFYwo/FUg0bCoaqQiYW5VlYOR8MFfGtiVc0ZBlUNsw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current code does not contemplate scenarios were an allocation and free operation on the same pages do not handle it in the same amount at once. To give an example, page_alloc_exact(), where we will allocate a page of enough order to stafisfy the size request, but we will free the remainings right away. In the above example, we will increment the stack_record refcount only once, but we will decrease it the same number of times as number of unused pages we have to free. This will lead to a warning because of refcount imbalance. Fix this by recording the number of base pages in the refcount field. Reported-by: syzbot+41bbfdb8d41003d12c0f@syzkaller.appspotmail.com Closes: https://lore.kernel.org/linux-mm/00000000000090e8ff0613eda0e5@google.com Fixes: 217b2119b9e2 ("mm,page_owner: implement the tracking of the stacks count") Signed-off-by: Oscar Salvador Reviewed-by: Vlastimil Babka --- Documentation/mm/page_owner.rst | 73 +++++++++++++++++---------------- mm/page_owner.c | 34 ++++++++------- 2 files changed, 58 insertions(+), 49 deletions(-) diff --git a/Documentation/mm/page_owner.rst b/Documentation/mm/page_owner.rst index 0d0334cd5179..3a45a20fc05a 100644 --- a/Documentation/mm/page_owner.rst +++ b/Documentation/mm/page_owner.rst @@ -24,10 +24,10 @@ fragmentation statistics can be obtained through gfp flag information of each page. It is already implemented and activated if page owner is enabled. Other usages are more than welcome. -It can also be used to show all the stacks and their outstanding -allocations, which gives us a quick overview of where the memory is going -without the need to screen through all the pages and match the allocation -and free operation. +It can also be used to show all the stacks and their current number of +allocated base pages, which gives us a quick overview of where the memory +is going without the need to screen through all the pages and match the +allocation and free operation. page owner is disabled by default. So, if you'd like to use it, you need to add "page_owner=on" to your boot cmdline. If the kernel is built @@ -75,42 +75,45 @@ Usage cat /sys/kernel/debug/page_owner_stacks/show_stacks > stacks.txt cat stacks.txt - prep_new_page+0xa9/0x120 - get_page_from_freelist+0x7e6/0x2140 - __alloc_pages+0x18a/0x370 - new_slab+0xc8/0x580 - ___slab_alloc+0x1f2/0xaf0 - __slab_alloc.isra.86+0x22/0x40 - kmem_cache_alloc+0x31b/0x350 - __khugepaged_enter+0x39/0x100 - dup_mmap+0x1c7/0x5ce - copy_process+0x1afe/0x1c90 - kernel_clone+0x9a/0x3c0 - __do_sys_clone+0x66/0x90 - do_syscall_64+0x7f/0x160 - entry_SYSCALL_64_after_hwframe+0x6c/0x74 - stack_count: 234 + post_alloc_hook+0x177/0x1a0 + get_page_from_freelist+0xd01/0xd80 + __alloc_pages+0x39e/0x7e0 + allocate_slab+0xbc/0x3f0 + ___slab_alloc+0x528/0x8a0 + kmem_cache_alloc+0x224/0x3b0 + sk_prot_alloc+0x58/0x1a0 + sk_alloc+0x32/0x4f0 + inet_create+0x427/0xb50 + __sock_create+0x2e4/0x650 + inet_ctl_sock_create+0x30/0x180 + igmp_net_init+0xc1/0x130 + ops_init+0x167/0x410 + setup_net+0x304/0xa60 + copy_net_ns+0x29b/0x4a0 + create_new_namespaces+0x4a1/0x820 + nr_base_pages: 16 ... ... echo 7000 > /sys/kernel/debug/page_owner_stacks/count_threshold cat /sys/kernel/debug/page_owner_stacks/show_stacks> stacks_7000.txt cat stacks_7000.txt - prep_new_page+0xa9/0x120 - get_page_from_freelist+0x7e6/0x2140 - __alloc_pages+0x18a/0x370 - alloc_pages_mpol+0xdf/0x1e0 - folio_alloc+0x14/0x50 - filemap_alloc_folio+0xb0/0x100 - page_cache_ra_unbounded+0x97/0x180 - filemap_fault+0x4b4/0x1200 - __do_fault+0x2d/0x110 - do_pte_missing+0x4b0/0xa30 - __handle_mm_fault+0x7fa/0xb70 - handle_mm_fault+0x125/0x300 - do_user_addr_fault+0x3c9/0x840 - exc_page_fault+0x68/0x150 - asm_exc_page_fault+0x22/0x30 - stack_count: 8248 + post_alloc_hook+0x177/0x1a0 + get_page_from_freelist+0xd01/0xd80 + __alloc_pages+0x39e/0x7e0 + alloc_pages_mpol+0x22e/0x490 + folio_alloc+0xd5/0x110 + filemap_alloc_folio+0x78/0x230 + page_cache_ra_order+0x287/0x6f0 + filemap_get_pages+0x517/0x1160 + filemap_read+0x304/0x9f0 + xfs_file_buffered_read+0xe6/0x1d0 [xfs] + xfs_file_read_iter+0x1f0/0x380 [xfs] + __kernel_read+0x3b9/0x730 + kernel_read_file+0x309/0x4d0 + __do_sys_finit_module+0x381/0x730 + do_syscall_64+0x8d/0x150 + entry_SYSCALL_64_after_hwframe+0x62/0x6a + nr_base_pages: 20824 ... cat /sys/kernel/debug/page_owner > page_owner_full.txt diff --git a/mm/page_owner.c b/mm/page_owner.c index 52d1ced0b57f..5df0d6892bdc 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -196,7 +196,8 @@ static void add_stack_record_to_list(struct stack_record *stack_record, spin_unlock_irqrestore(&stack_list_lock, flags); } -static void inc_stack_record_count(depot_stack_handle_t handle, gfp_t gfp_mask) +static void inc_stack_record_count(depot_stack_handle_t handle, gfp_t gfp_mask, + int nr_base_pages) { struct stack_record *stack_record = __stack_depot_get_stack_record(handle); @@ -217,15 +218,20 @@ static void inc_stack_record_count(depot_stack_handle_t handle, gfp_t gfp_mask) /* Add the new stack_record to our list */ add_stack_record_to_list(stack_record, gfp_mask); } - refcount_inc(&stack_record->count); + refcount_add(nr_base_pages, &stack_record->count); } -static void dec_stack_record_count(depot_stack_handle_t handle) +static void dec_stack_record_count(depot_stack_handle_t handle, + int nr_base_pages) { struct stack_record *stack_record = __stack_depot_get_stack_record(handle); - if (stack_record) - refcount_dec(&stack_record->count); + if (!stack_record) + return; + + if (refcount_sub_and_test(nr_base_pages, &stack_record->count)) + pr_warn("%s: refcount went to 0 for %u handle\n", __func__, + handle); } static inline void __update_page_owner_handle(struct page_ext *page_ext, @@ -306,7 +312,7 @@ void __reset_page_owner(struct page *page, unsigned short order) * the machinery is not ready yet, we cannot decrement * their refcount either. */ - dec_stack_record_count(alloc_handle); + dec_stack_record_count(alloc_handle, 1 << order); } noinline void __set_page_owner(struct page *page, unsigned short order, @@ -325,7 +331,7 @@ noinline void __set_page_owner(struct page *page, unsigned short order, current->pid, current->tgid, ts_nsec, current->comm); page_ext_put(page_ext); - inc_stack_record_count(handle, gfp_mask); + inc_stack_record_count(handle, gfp_mask, 1 << order); } void __set_page_owner_migrate_reason(struct page *page, int reason) @@ -872,11 +878,11 @@ static void *stack_next(struct seq_file *m, void *v, loff_t *ppos) return stack; } -static unsigned long page_owner_stack_threshold; +static unsigned long page_owner_pages_threshold; static int stack_print(struct seq_file *m, void *v) { - int i, stack_count; + int i, nr_base_pages; struct stack *stack = v; unsigned long *entries; unsigned long nr_entries; @@ -887,14 +893,14 @@ static int stack_print(struct seq_file *m, void *v) nr_entries = stack_record->size; entries = stack_record->entries; - stack_count = refcount_read(&stack_record->count) - 1; + nr_base_pages = refcount_read(&stack_record->count) - 1; - if (stack_count < 1 || stack_count < page_owner_stack_threshold) + if (nr_base_pages < 1 || nr_base_pages < page_owner_pages_threshold) return 0; for (i = 0; i < nr_entries; i++) seq_printf(m, " %pS\n", (void *)entries[i]); - seq_printf(m, "stack_count: %d\n\n", stack_count); + seq_printf(m, "nr_base_pages: %d\n\n", nr_base_pages); return 0; } @@ -924,13 +930,13 @@ static const struct file_operations page_owner_stack_operations = { static int page_owner_threshold_get(void *data, u64 *val) { - *val = READ_ONCE(page_owner_stack_threshold); + *val = READ_ONCE(page_owner_pages_threshold); return 0; } static int page_owner_threshold_set(void *data, u64 val) { - WRITE_ONCE(page_owner_stack_threshold, val); + WRITE_ONCE(page_owner_pages_threshold, val); return 0; } From patchwork Tue Mar 26 06:30:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 13603618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7DFBC54E67 for ; Tue, 26 Mar 2024 06:29:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AEB86B009B; Tue, 26 Mar 2024 02:29:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 75D0C6B009C; Tue, 26 Mar 2024 02:29:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D7916B009D; Tue, 26 Mar 2024 02:29:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4781E6B009B for ; Tue, 26 Mar 2024 02:29:27 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1172D1C081E for ; Tue, 26 Mar 2024 06:29:27 +0000 (UTC) X-FDA: 81938213574.14.17F8BFC Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf12.hostedemail.com (Postfix) with ESMTP id DBB6140010 for ; Tue, 26 Mar 2024 06:29:24 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=p9jMLvOo; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=wcN3jXol; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=p9jMLvOo; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=wcN3jXol; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf12.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711434565; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=N3hPM+dJZjrUpvPFcXGaUP6KRSTdfQGjfQ6el+a/Oj0=; b=f6fsNVVoMyBmPZ018/e7XdEJtMgpIX6x0hpmvIt+tQXFidrxGqidY2mr7oQcwll+zA3R1O 3Fc2UUNFlkyTVTmI38K/AmUuWY7IOvnpiSp73ISc6M+Q1gD7WG4Q7qOtUCM7BWBdB7zbzu LeTfNVd7MgN+K2tAv9cjGckIWKM90ec= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=p9jMLvOo; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=wcN3jXol; dkim=pass header.d=suse.de header.s=susede2_rsa header.b=p9jMLvOo; dkim=pass header.d=suse.de header.s=susede2_ed25519 header.b=wcN3jXol; dmarc=pass (policy=none) header.from=suse.de; spf=pass (imf12.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711434565; a=rsa-sha256; cv=none; b=053F4ynknHZ/6QTN/VmMR+s65PBfZwYzvhCG4gzDqNEOeZDKpL/gb4HGoUMRmfb2wlHSQj 9e9BiEMGQmYxZF8fXsLLkWzSMMM9fHZhEcEO0Bl1XGi359K4tiRM77heoDLPB4sLgv5l9m DXGeda2plvzReduZdyn/3uoftqqcpD8= Received: from imap2.dmz-prg2.suse.org (imap2.dmz-prg2.suse.org [10.150.64.98]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7FB4C5D1D0; Tue, 26 Mar 2024 06:29:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1711434563; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N3hPM+dJZjrUpvPFcXGaUP6KRSTdfQGjfQ6el+a/Oj0=; b=p9jMLvOo6GG9vIewO6sQfRTZ4Mni1Sdahpm3nMkhEbh16q6iJRNISCm+ZrBTYFRWpY5YcT Lqe8NCKu2D1MHp2522H0z8fCEaKmnc3/42wskN9RWdtmSdi8ocsSRMxp3FSWIIq2rdtKNQ LGG0D7tPy5FcURCuhqUWr2cxLFOah9M= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1711434563; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N3hPM+dJZjrUpvPFcXGaUP6KRSTdfQGjfQ6el+a/Oj0=; b=wcN3jXol/cm7dMUuI/mYHwOilVrIFUsqPKVsZqU5Il++LJ0ABOWuEOeQWXzgRWynNrtLwp TZDUMr4PFDXkmGBw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1711434563; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N3hPM+dJZjrUpvPFcXGaUP6KRSTdfQGjfQ6el+a/Oj0=; b=p9jMLvOo6GG9vIewO6sQfRTZ4Mni1Sdahpm3nMkhEbh16q6iJRNISCm+ZrBTYFRWpY5YcT Lqe8NCKu2D1MHp2522H0z8fCEaKmnc3/42wskN9RWdtmSdi8ocsSRMxp3FSWIIq2rdtKNQ LGG0D7tPy5FcURCuhqUWr2cxLFOah9M= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1711434563; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N3hPM+dJZjrUpvPFcXGaUP6KRSTdfQGjfQ6el+a/Oj0=; b=wcN3jXol/cm7dMUuI/mYHwOilVrIFUsqPKVsZqU5Il++LJ0ABOWuEOeQWXzgRWynNrtLwp TZDUMr4PFDXkmGBw== Received: from imap2.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap2.dmz-prg2.suse.org (Postfix) with ESMTPS id 01EA613215; Tue, 26 Mar 2024 06:29:22 +0000 (UTC) Received: from dovecot-director2.suse.de ([10.150.64.162]) by imap2.dmz-prg2.suse.org with ESMTPSA id KGdpOUJrAmY2YQAAn2gu4w (envelope-from ); Tue, 26 Mar 2024 06:29:22 +0000 From: Oscar Salvador To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Michal Hocko , Vlastimil Babka , Marco Elver , Andrey Konovalov , Alexander Potapenko , Oscar Salvador Subject: [PATCH v3 3/3] mm,page_owner: Fix accounting of pages when migrating Date: Tue, 26 Mar 2024 07:30:36 +0100 Message-ID: <20240326063036.6242-4-osalvador@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240326063036.6242-1-osalvador@suse.de> References: <20240326063036.6242-1-osalvador@suse.de> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 7yjr9q5pym4zwfexsrbecc16z1q79mix X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DBB6140010 X-HE-Tag: 1711434564-943215 X-HE-Meta: U2FsdGVkX1+Bv/5c3hw31WWQU3QsN3amQqtRxJG1Vf38pwqcbqBJccb85WtyeNfjaC1mGv45os2RUW0ymQQ4oUkUh6bGs5j8gvx/4rUSkcKL9Sh8cUHszCkAbldhMOg2mMW9rUMFHg8LNeZzBBqk2Hjscocj8pKhmPh730P6Ym+Vk1Uq+zLcdyCPL6+d0/4cAQy3uZycWIVJOVmAy3VOm3EAM/AznahuqvacwMvycr0ABpc21UKcrbjzDCZaNhAzLbDiMnf3IvRBKPGf1lBkQyFo0keyreViJ2bmj4VhqnocOJW4WhvGGtIbwtrO1+1aQSfqOlmhBm6wMdU2uf7e+1Bf36L7xhFyDKn3EOclKjjv7QSluhBxeAHAu3RPmk3xW7pjNo2WEXZoBRa1OPET/cyAQManbf65H8VOlI50JQMautUeT4K99GoPzzeYGzPeBBKsrSdHIY9vlTZc/uindjMePxyh/agZrolaQn5WDlFpqgHbcuqxDH/pkmvj69TH/MgWRnIUpw6FaLZujLw74YC/HYhInsY5roifwxZNJ0EKRB3k6qyEs3c4+M9h5d5e17DQ170LbuGTvqkzwH68aeqRWLDFmrjbiaWF+8orLK12rbF5sJDQaAwfBWYIhrRjihLNDwNM2e5ZpB0Frw/y2+aVT3OWf/cNTLfZGNaDwt3pEYVIMDUl1SJ/kvVyM48O4E0xn8eRmwbiIq0ajenIyyiYW0/bzwKBbSoloMvBR8GucpeofYFyg/rXoZf1vY/+4G+xNvIZ5hGKHGYMKFwuaCIQbDmQ7Xn09XaqzI508UwuMphrOFY+bQW5xeMM60HH/v9h5z0TPKmJi+5IRbwEe62Q6sEeMXxSId1kT52ZEE6K6VsAn21Y4klM2XIvrZPkZ+pYMG2T16hh+O6hYPUSyWXAkoh6Ww6K3TL1RCnRosO+zQcxwMMCQssuiMTso2ul+NzrAK5YIklxEjLrSE5 qTPs2wMS pZzq93PdKyUsh4h7gem2G2Q0h2QBpvjCHMAWk+saiACTFy+8sql+1kuWABh1PORBjiyZB5sc+/ee4Khuq9x4M9RDXe71qy2Rrec3SzDyaD6MsOBXCb8/iFLC7Zn80/zO+EUyTDcM/ABbFJfFF8sSRy//z4mqLmgSlbYGbNtg30bALnZaFhtnzEy04U1xi0JB9x3Ayf8EhTiJzEIDiLxDQ5pYktEFfM7TTwCo62/fDJ4h8UF4WhQj3fNxE7j4hMTIujCSq X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Upon migration, new allocated pages are being given the handle of the old pages. This is problematic because it means that for the stack which allocated the old page, we will be substracting the old page + the new one when that page is freed, creating an accounting imbalance. There is an interest in keeping it that way, as otherwise the output will biased towards migration stacks should those operations occur often, but that is not really helpful. The link from the new page to the old stack is being performed by calling __update_page_owner_handle() in __folio_copy_owner(). The only thing that is left is to link the migrate stack to the old page, so the old page will be subtracted from the migrate stack, avoiding by doing so any possible imbalance. Fixes: 217b2119b9e2 ("mm,page_owner: implement the tracking of the stacks count") Signed-off-by: Oscar Salvador Reviewed-by: Vlastimil Babka --- mm/page_owner.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/mm/page_owner.c b/mm/page_owner.c index 5df0d6892bdc..b4476f45b376 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -366,9 +366,12 @@ void __split_page_owner(struct page *page, int old_order, int new_order) void __folio_copy_owner(struct folio *newfolio, struct folio *old) { + int i; struct page_ext *old_ext; struct page_ext *new_ext; struct page_owner *old_page_owner; + struct page_owner *new_page_owner; + depot_stack_handle_t migrate_handle; old_ext = page_ext_get(&old->page); if (unlikely(!old_ext)) @@ -381,6 +384,8 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) } old_page_owner = get_page_owner(old_ext); + new_page_owner = get_page_owner(new_ext); + migrate_handle = new_page_owner->handle; __update_page_owner_handle(new_ext, old_page_owner->handle, old_page_owner->order, old_page_owner->gfp_mask, old_page_owner->last_migrate_reason, @@ -395,6 +400,16 @@ void __folio_copy_owner(struct folio *newfolio, struct folio *old) old_page_owner->free_pid, old_page_owner->free_tgid, old_page_owner->free_ts_nsec); + /* + * We linked the original stack to the new folio, we need to do the same + * for the new one and the old folio otherwise there will be an imbalance + * when subtracting those pages from the stack. + */ + for (i = 0; i < (1 << new_page_owner->order); i++) { + old_page_owner->handle = migrate_handle; + old_ext = page_ext_next(old_ext); + old_page_owner = get_page_owner(old_ext); + } page_ext_put(new_ext); page_ext_put(old_ext);