From patchwork Wed Feb 28 05:11:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Chang X-Patchwork-Id: 13574865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D10C1C47DD9 for ; Wed, 28 Feb 2024 05:11:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 225976B014D; Wed, 28 Feb 2024 00:11:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D4B56B014F; Wed, 28 Feb 2024 00:11:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09C666B0150; Wed, 28 Feb 2024 00:11:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EE6426B014D for ; Wed, 28 Feb 2024 00:11:44 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B47BEC0AF5 for ; Wed, 28 Feb 2024 05:11:44 +0000 (UTC) X-FDA: 81840040128.18.AF0113D Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf02.hostedemail.com (Postfix) with ESMTP id EBC7280014 for ; Wed, 28 Feb 2024 05:11:42 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=eVcFQB02; spf=pass (imf02.hostedemail.com: domain of 3jsDeZQoKCL8wnhmfwi3hhlttlqj.htrqnsz2-rrp0fhp.twl@flex--richardycc.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3jsDeZQoKCL8wnhmfwi3hhlttlqj.htrqnsz2-rrp0fhp.twl@flex--richardycc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709097103; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uqML4vY0pGUSQ+ssAelgPTzgnCg/ot5G0/OjWps3luM=; b=R9ehjWxkA50pRo7XEauGpGTwmM/f8T8RNRkf+umUNsuvQVipu9+OQA4q5j73rtf3QjwXuW NI6r+xpatX40EVVP05TOVtnPNbhnhw2ruGYUqfmXxKLdQZiAWFFIe33GJnFi7ER5xyz5BS O254o0vNLgm6XRU5+81yeNYv5UZ9zoY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709097103; a=rsa-sha256; cv=none; b=N/OXMKFNT6ENdLXcoLsfScpNpuLvhatYDGlK0WycaL1212YqpnxgqXXicpkANmoY3SvNpZ flSE8pwsvznZEcesyKIeBcps1IwgHkgA+1roEUa0Ds4OEfQOUQIqL8T/Jp3p0cfO12I4fS N2roat+YopUJI0ozB179UBU204MAMKI= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=eVcFQB02; spf=pass (imf02.hostedemail.com: domain of 3jsDeZQoKCL8wnhmfwi3hhlttlqj.htrqnsz2-rrp0fhp.twl@flex--richardycc.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3jsDeZQoKCL8wnhmfwi3hhlttlqj.htrqnsz2-rrp0fhp.twl@flex--richardycc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-608ab197437so82232267b3.1 for ; Tue, 27 Feb 2024 21:11:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709097102; x=1709701902; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=uqML4vY0pGUSQ+ssAelgPTzgnCg/ot5G0/OjWps3luM=; b=eVcFQB02t5OchpF72yOU59xzEhR/16PUNHru7v479ZHgO5PU8gQ2y3LD2Potlcc80I 7ENVZdYQ0+ac+35Lb4fHBmZcPVKendtIATFFdcSyqtjXfNgh6lbFuNapBYbcl9stqpQJ M3QO6n5z7DtKYJ5odTdm0AOA8ClBTI2QEmlkXu4i00nUUdzKWpUYAi6NxL3pDZCUqwrS JGOq5iEeAJkCr0xlkU00O1h/mhAuitiW1F/lLEn5AULr3Oz5aV1FYrN+j7LFKQ1pl1cI Nd5KeD0mY6DaTHtW1f8kaEDVk/aPf8n5asQ2ViIYrjNChlrf2aM1V/uGR4G4EimEOv0O /gWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709097102; x=1709701902; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=uqML4vY0pGUSQ+ssAelgPTzgnCg/ot5G0/OjWps3luM=; b=OZ4p2Lww/nliAPvQ894HTKqdn3GDVYYMl+mZ3FJ4UJHu+84YQ3yW0DtJzzSjxoh53S joPbDpHP0O3t7wEL/ySmGY6HYv9gzVl1ZiOMDG21709viCaxEAf+/kuvPjv2nxvV0f13 zhlxP+yauMD6vzasEvFKPmmzNMOuI8bubZj0NzMqsu0K73uvTPZLaDaw7q/gtYlNPZqh SrfguxRAq8WCRXK4IkG2APrnq5Bp1MRIg0qwJBtue4jnh5zNORaCxpWCkMzoH6cHWDC/ E20WiWX1tzuXzWYWeIlQZVulFgSEA8TSJSwYFcmNqK35u2qPbvCWZfUB0N/ygCK+qbX0 DTBg== X-Forwarded-Encrypted: i=1; AJvYcCU5WKEWChbm6LjpUVfAMjiPHdfkL0Ku+Qma2zk0WvKcZ8Lb9nXNIhgH67Ww6TZMB66S++nnyMy7wxPewKV0cUJxI08= X-Gm-Message-State: AOJu0YyWcrSgReFniQRLbBq1/dG1SNfqVRk/nGjrZokPK0Kcr73mn+HG BOngY21aOqKdj4FcTJ0LDEbsiDTqcZo1XITkZBSydVIN2NYDLo+GvPdW9qnJd+rWLDIUFag9aED bZzKcb0if/jL4XNRo X-Google-Smtp-Source: AGHT+IHyzMYAk41ghRY9BmWyJ7wM13H2JKhk13gZrcP4KgPdiL8ORxRnV4ac5QW5ZsOurSY8RayignJdNKweROeL X-Received: from richardycc.c.googlers.com ([fda3:e722:ac3:cc00:3:22c1:c0a8:f5f]) (user=richardycc job=sendgmr) by 2002:a05:690c:88:b0:609:247a:bdc5 with SMTP id be8-20020a05690c008800b00609247abdc5mr955171ywb.4.1709097102017; Tue, 27 Feb 2024 21:11:42 -0800 (PST) Date: Wed, 28 Feb 2024 05:11:17 +0000 In-Reply-To: <20240226120728.3f495fa7@gandalf.local.home> Mime-Version: 1.0 References: <20240226120728.3f495fa7@gandalf.local.home> X-Mailer: git-send-email 2.44.0.rc1.240.g4c46232300-goog Message-ID: <20240228051127.2859472-1-richardycc@google.com> Subject: [PATCH v2] mm: add alloc_contig_migrate_range allocation statistics From: Richard Chang To: rostedt@goodmis.org, mhiramat@kernel.org, mathieu.desnoyers@efficios.com, akpm@linux-foundation.org, richardycc@google.com Cc: liumartin@google.com, surenb@google.com, minchan@kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org X-Stat-Signature: 4ywnbrytsy8o3mt44zb3trgkyebxhgf7 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EBC7280014 X-Rspam-User: X-HE-Tag: 1709097102-871994 X-HE-Meta: U2FsdGVkX19Bn0KbCpwcaJWibhYH3rjT97IAd4e5aWkJrW5/v/Q3l3NR6rjbwu8QOYEHKijSjDGyKfGPG/UlGkuR2dmgWbQmqApkIr/ZO6ftjLvST2zYcmpYtThGNwmGfWN9MLrJIwGQqp4dJ58OkFpkuLbQub6nra+dFqAWsBiJLSGyD/CahuCLfivpW/+n7vx0A38d6wt63KeEF721eA7ML+95KPpjDHTfIaEWLR56g3wswkon5TC35hUWXea8+CK9ovWuvKnoTyY4DOx5pQvTlNcnMfP531uLn4ih3T3TjU9vClhGqN2l3R1Ip8cG+01dO8LVqSLqnFrygcQmAiqMBQ24TXPCyLcR4yyZ3aOhc735Ic0a1buXugR0/XT3Kj2zJ4k5NwieFnoo11mW0KCmw6xX2wkHgsfJQl+V7Qc3BxdYhihWUAsy3OVUbnfJ5crmTLsDVLvCB3mOpPVSv9v7+CN6VZ9zhEkmLRNfB3LqT7y7ZKAnMlL4w0T1xzwcppCx88vkvq/x/M4qqxjwuDweNNJiPE7E75SEnWObbITozbGSWNbsoXh4YBkgDtrZEMawJCOy6ePljEeNKHYKFrB7VST8Iy0wPP9jtKiPUi8ed9dhXqitfHTMJ88bN3anIi2XRUqK7V8BsHiQZEgZSrEBUEUbXJfbnn9JtVrsg+e8gNG1dDNWlkh22lx3mE/IZsmlAaxylzO2eyTCZhLSFXMnyOie65Rkroz9GT5uYH8eJg94oqC2VR3deWBIWYAG3gXm1RFS8P7Mfsd0BejBI8W7QkKT+P7GreLemfZpj9jISG7wNkJvI6bauhCX4BP0fSJC2Y3l/L1OoLIftXSDRUPJlxQv1lG63QnR94ymyrVJOolt8701puuiq/VTGCdSwCL/NuA0aAgYNA59AWTIW5ULfwXr95n/fR/+QzWg+City41usJsRedAtZ4j2g7KOIoUP7/cEu8ebWgAfQV4 72UGAV0G gta99k8prgfRBYGzDhKNpVdoe1KuEDU/0LJ3IA0DPmn+oalsn/as0+2eNPRmvRre2GdwKo6cIGFUPoxbqDack3zrw4woa651DbUGKv3F5qKbOnLtgpA+kljjgeV1ECcwISwAIEZQ0YT1Tj+RrMmb3NPC59iv7jia8JCopydfi40w6r9hFneanxIrHImJWNSz9RdtzMAIxsLDVcUkPr/hTZXkAQAEHS83ot9L+7WKWaIol53JqU4WmuyeVdn1Q325w4LYqrn//PVwNKSoHM52lyxz0Bx3H4I3c2yzXNkjMBPL4jlr0F1XHu+SR3l1hNd3L4m0uNG8fgNuOHrHD3AWvjmEsaY8wfXTNm7OeWfix3J48a9y1w/bITrJfmxqxxk1DnFdL5Y/gN/I6xIt02M3YJY44cLH4WzTlFzFOgtwd4nSpJ6zJmOejEqFqT70jR23lxleF6OT5mRq63GCSVTr9X/8KpQdfF/F/qDECa4qtF/RVTdygfWD+4x5Lq4ZmrteyRMGiz8IIgWeoYXmdTS2pxCjmrnn4/BXjcmiXlZwe/oeho5qjwnrRjwF950pJk7F7xKcTHAiIz1/NpHNOm6fEjPHgh+gK5IJ+cKHMHV41ZHVsNP/3nabmtiNBslfNc24+YxskxJft9hkuFLusmvuIcKrONM8yTZJzm9D+qANKdG4AMPydzXrfh0ca/7Ym41DCT1by9p2PRCAiFrGLF2ob5GGlLjH8JaumSGrYZPaYBqzMNHw1v8O8Aseq7g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: alloc_contig_migrate_range has every information to be able to understand big contiguous allocation latency. For example, how many pages are migrated, how many times they were needed to unmap from page tables. This patch adds the trace event to collect the allocation statistics. In the field, it was quite useful to understand CMA allocation latency. Signed-off-by: Richard Chang Reviewed-by: Steven Rostedt (Google) --- * from v1 - https://lore.kernel.org/linux-mm/20240226100045.2083962-1-richardycc@google.com/ * Move the trace event int field to the end of the longs - rostedt * Do the calculation only when tracing is enabled - rostedt include/trace/events/kmem.h | 38 +++++++++++++++++++++++++++++++++++++ mm/internal.h | 3 ++- mm/page_alloc.c | 32 ++++++++++++++++++++++++++----- mm/page_isolation.c | 2 +- 4 files changed, 68 insertions(+), 7 deletions(-) diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index 58688768ef0f..6e62cc64cd92 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -304,6 +304,44 @@ TRACE_EVENT(mm_page_alloc_extfrag, __entry->change_ownership) ); +TRACE_EVENT(mm_alloc_contig_migrate_range_info, + + TP_PROTO(unsigned long start, + unsigned long end, + unsigned long nr_migrated, + unsigned long nr_reclaimed, + unsigned long nr_mapped, + int migratetype), + + TP_ARGS(start, end, nr_migrated, nr_reclaimed, nr_mapped, migratetype), + + TP_STRUCT__entry( + __field(unsigned long, start) + __field(unsigned long, end) + __field(unsigned long, nr_migrated) + __field(unsigned long, nr_reclaimed) + __field(unsigned long, nr_mapped) + __field(int, migratetype) + ), + + TP_fast_assign( + __entry->start = start; + __entry->end = end; + __entry->nr_migrated = nr_migrated; + __entry->nr_reclaimed = nr_reclaimed; + __entry->nr_mapped = nr_mapped; + __entry->migratetype = migratetype; + ), + + TP_printk("start=0x%lx end=0x%lx migratetype=%d nr_migrated=%lu nr_reclaimed=%lu nr_mapped=%lu", + __entry->start, + __entry->end, + __entry->migratetype, + __entry->nr_migrated, + __entry->nr_reclaimed, + __entry->nr_mapped) +); + /* * Required for uniquely and securely identifying mm in rss_stat tracepoint. */ diff --git a/mm/internal.h b/mm/internal.h index f309a010d50f..e114c647e278 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -537,7 +537,8 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long low_pfn, unsigned long end_pfn); int __alloc_contig_migrate_range(struct compact_control *cc, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, + int migratetype); /* Free whole pageblock and set its migration type to MIGRATE_CMA. */ void init_cma_reserved_pageblock(struct page *page); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 150d4f23b010..da169fa20d8e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6219,9 +6219,14 @@ static void alloc_contig_dump_pages(struct list_head *page_list) } } -/* [start, end) must belong to a single zone. */ +/* + * [start, end) must belong to a single zone. + * @migratetype: using migratetype to filter the type of migration in + * trace_mm_alloc_contig_migrate_range_info. + */ int __alloc_contig_migrate_range(struct compact_control *cc, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, + int migratetype) { /* This function is based on compact_zone() from compaction.c. */ unsigned int nr_reclaimed; @@ -6232,6 +6237,10 @@ int __alloc_contig_migrate_range(struct compact_control *cc, .nid = zone_to_nid(cc->zone), .gfp_mask = GFP_USER | __GFP_MOVABLE | __GFP_RETRY_MAYFAIL, }; + struct page *page; + unsigned long total_mapped = 0; + unsigned long total_migrated = 0; + unsigned long total_reclaimed = 0; lru_cache_disable(); @@ -6257,9 +6266,18 @@ int __alloc_contig_migrate_range(struct compact_control *cc, &cc->migratepages); cc->nr_migratepages -= nr_reclaimed; + if (trace_mm_alloc_contig_migrate_range_info_enabled()) { + total_reclaimed += nr_reclaimed; + list_for_each_entry(page, &cc->migratepages, lru) + total_mapped += page_mapcount(page); + } + ret = migrate_pages(&cc->migratepages, alloc_migration_target, NULL, (unsigned long)&mtc, cc->mode, MR_CONTIG_RANGE, NULL); + if (trace_mm_alloc_config_migrate_range_info_enabled() && !ret) + total_migrated += cc->nr_migratepages; + /* * On -ENOMEM, migrate_pages() bails out right away. It is pointless * to retry again over this error, so do the same here. @@ -6273,9 +6291,13 @@ int __alloc_contig_migrate_range(struct compact_control *cc, if (!(cc->gfp_mask & __GFP_NOWARN) && ret == -EBUSY) alloc_contig_dump_pages(&cc->migratepages); putback_movable_pages(&cc->migratepages); - return ret; } - return 0; + + trace_mm_alloc_contig_migrate_range_info(start, end, migratetype, + total_migrated, + total_reclaimed, + total_mapped); + return (ret < 0) ? ret : 0; } /** @@ -6355,7 +6377,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, * allocated. So, if we fall through be sure to clear ret so that * -EBUSY is not accidentally used or returned to caller. */ - ret = __alloc_contig_migrate_range(&cc, start, end); + ret = __alloc_contig_migrate_range(&cc, start, end, migratetype); if (ret && ret != -EBUSY) goto done; ret = 0; diff --git a/mm/page_isolation.c b/mm/page_isolation.c index cd0ea3668253..a5c8fa4c2a75 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -434,7 +434,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, } ret = __alloc_contig_migrate_range(&cc, head_pfn, - head_pfn + nr_pages); + head_pfn + nr_pages, page_mt); /* * restore the page's migratetype so that it can