From patchwork Mon Feb 26 10:00:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Chang X-Patchwork-Id: 13571787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 670ECC48BF6 for ; Mon, 26 Feb 2024 10:01:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE4F2940029; Mon, 26 Feb 2024 05:01:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C93BD940014; Mon, 26 Feb 2024 05:01:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5C21940029; Mon, 26 Feb 2024 05:01:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A53C7940014 for ; Mon, 26 Feb 2024 05:01:52 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8286D1203F4 for ; Mon, 26 Feb 2024 10:01:52 +0000 (UTC) X-FDA: 81833513664.21.A13BB1F Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf13.hostedemail.com (Postfix) with ESMTP id E109A20021 for ; Mon, 26 Feb 2024 10:01:50 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BIbcbi+n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3jWHcZQoKCPYpgafYpbwaaemmejc.amkjglsv-kkitYai.mpe@flex--richardycc.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3jWHcZQoKCPYpgafYpbwaaemmejc.amkjglsv-kkitYai.mpe@flex--richardycc.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708941711; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=gyI/MoWdVSABhMtiVRAkTZHZn+lW5oiA7R70q9bef8k=; b=A//KzIAfAi46SmHKMoYVKt2uZaVnuybPnfGl22kk0OT0jn9odPUvq4F4N1ul76kTNBx1o3 xsQOPUkJ/4neTcnSw6Cj5oSzZ7kEGkoJ0AqlrhqlIUalu7jf9eA4kb220tohIRuI+Y3O/o P0BaQ2elz2+QamFSeQkuS7fdDlBh+vc= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=BIbcbi+n; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3jWHcZQoKCPYpgafYpbwaaemmejc.amkjglsv-kkitYai.mpe@flex--richardycc.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3jWHcZQoKCPYpgafYpbwaaemmejc.amkjglsv-kkitYai.mpe@flex--richardycc.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708941711; a=rsa-sha256; cv=none; b=K5QfznIlvrN8LuivwITBC7VdjXEZujcQiKpofm4LEGIc8/Uc7TrMywicqtpYeHXDmyzIPu xuSjJmN+anIjIuJqux510NyzQje7sqFOHGn38p0bq26jLvrYAVTWPZPcjSMYF8QmJiTtyG 6U9jjvTekqioPymLPXMoVOgQT9HtMXA= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-6e4e7d360f4so1697363b3a.3 for ; Mon, 26 Feb 2024 02:01:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1708941710; x=1709546510; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=gyI/MoWdVSABhMtiVRAkTZHZn+lW5oiA7R70q9bef8k=; b=BIbcbi+nfbTVVtLxwxOb/IQBr00hll7M9H9Znql/xcba06bdvf2sRK+kvigdQ8XqmL SUNPRV2hE03GWqnxcqb/IzZLMY+qk48O45xADtXLO6IIgJHqIya4/AFgDJm4owuuejud i2Bg6lnSkcbqHykmMAYnNHwJJQ3Nab1XjiZu2Jp8YHFpeJErySPW9Ohm2EOgOqS4jxuI JCGqNlvcZjxb7gskPDIKo0wzT2c5HClIIsaQWkjgAB6k6Ig9IZYaj+F89J3g3T7irTWc BhLRUudGm8om227G9jLcOC/r7kvSQeOYatSSEu1rFbtdHzXJ9C5atoOFALRNiwwuYGda qWJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708941710; x=1709546510; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gyI/MoWdVSABhMtiVRAkTZHZn+lW5oiA7R70q9bef8k=; b=poqPd9XSbVPVha0+hAdTWoEk/wgSE6ZS6a++jXxjr64xkt3mkjuw6lnkdr5j1wzwVC zzpV0tuRkbrd3asNGbiQo+F/TDym4mlgPeucvwd1gilpXm5b7Wcp35QST1JnAUWragFL vTTBCluEkSzHk/xmK4O+G23YuwtByHfNILPk9j7AIMlO1CMBLY8yRU1yyEroYel3HdH+ 72c58sUE2cppk+5Iq6LMO3z6D7dAmS5MY3Xv+ZTdrRnHK46dPtsew+zdmegPeR3geALv HRT5aHi47UZBOONxe+6ha2Y5HgU0baDcYCpSBHlCR2FmE2K4wn0lm3S24utUBMVR8oCF UgcQ== X-Forwarded-Encrypted: i=1; AJvYcCUajkqcvhLtXIr5K0yR2Nqu3eW0U+uSBb48JZPQG+37I4xl9gB9o8H0P4nOyUSxQE3sjUxZ3xJgsCB46nosvaa2v5M= X-Gm-Message-State: AOJu0YxxBh0GM57xfANsJl3wh2zFQKPaaWuDQYu+SrrsLqa4bj7Ci5nE WXDiIZsoBUiXQOt+QYuK4kZae74erDk+l7UsEBcoB0f9HXOmNpY+wWbg40QZsEFaqK76GGeNh6V AL/lbK/wXLkbnnYDa X-Google-Smtp-Source: AGHT+IHsMsnGd+L6Ih3NO8l76Y1KOj9vYrg8gPa96vnLXi5uITU09PrnBtcNIlKstccydMSWJ4PxB/iLZvaiHzoW X-Received: from richardycc.c.googlers.com ([fda3:e722:ac3:cc00:3:22c1:c0a8:f5f]) (user=richardycc job=sendgmr) by 2002:aa7:88d4:0:b0:6e5:3590:5dc3 with SMTP id k20-20020aa788d4000000b006e535905dc3mr9037pff.4.1708941709522; Mon, 26 Feb 2024 02:01:49 -0800 (PST) Date: Mon, 26 Feb 2024 10:00:15 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.44.0.rc0.258.g7320e95886-goog Message-ID: <20240226100045.2083962-1-richardycc@google.com> Subject: [PATCH] mm: add alloc_contig_migrate_range allocation statistics From: Richard Chang To: rostedt@goodmis.org, mhiramat@kernel.org, mathieu.desnoyers@efficios.com, akpm@linux-foundation.org, richardycc@google.com Cc: liumartin@google.com, surenb@google.com, minchan@kernel.org, linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-mm@kvack.org X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E109A20021 X-Stat-Signature: ppcoxzqrnqcze8dydm5eh785bxtngcko X-Rspam-User: X-HE-Tag: 1708941710-940966 X-HE-Meta: U2FsdGVkX1/RRZsOk0IaqgL4OnSotLZj4wXmlFaUJeUXTcaM0I9MXFEkTt2a3iiQHkD/lOihAmSg2Ks4w+xO0UXDWiOYH++NHoYpx7pDAKLXN68hUK7TSwIsy73Z6HdWqW0zuii2eIvcWlR0ap9slQLHCouLvbMy8dRJHoN+D7WG14Gnv7LY6HyA1ORvrIXm36LzNLpefgwp6ueZ8ASutVnQS4ARgsGjfm2Jn4ofjaidT7XiNwgeuFKRfdOz04rcl0WgDwFZV24wwzx2pT+gC0eFqirv1aO4nd2TOZKgGfflitjWconmcH9SN2g/+q4F9uUzxc0PUEWrOsP8ti8BrfnZGmqv0/K4+LulKWNfI8lj5rcoDU7T7NDCdg1lGvAWXuEChfvdd9n/cFU6qIA1tH7bidCIRCtbUwxF/XN5DcjVO/CRsIK2gaQBtgB2/GgFxgbwlYC77tNYr1EHOU+7f25+9oPE5UQ11z2edNmtKlIjDI7RTzOxCEPg++81FpOT5WAnQpDlUHK/bygLzb88Q1XOPMtPzFXcWYsVW+GvCBwZj/ieuJ70d+h3+EfibgnXjODsoHfzOPuHchYXaD1qf57OsxJEwrcA90tUC6laI8p4rXtg3UjiBc3uKCpF717lqdOVRl7GgKWsyh2g2/FK5wJx9q6qBLi3VHw0EnmDywDVQ6RDW33WiyrbtxieFFZvmjv3QQdGcU1BQcciOnPpXRotFuc8upTsF3C282JrSOUGhPSc3PcnM7aerdf/pLnoYMmr+9PCYHl1gGdqwIbQ6aWqEO7rJHZMmSGoOq8E2Y4BtFXppsExqBP/pisnF4l59Ke09EF0rqIMI43Pxofp/vIBRTDBRXcDbHRNVmRg6KqnG1WDf/JEizBlFi3KSA3nVnaIG3CNM9YuephP0bDDyrT2JDouRBIgrNpk6mPHaVA/nNbO3WM7y2FLyBNf6HI5OyCU6ka27yngfzwMteP JeZDcH7B TuPG3jbI5l5GEBjC1/74T0SBiwkewW3LeHPEXVTwKlATTX0HklMS+GGOsUsu49ltnicJmCckpCQKYn1+6fhXVHnrBZEswHFcHJoOJhHyiEFF/jZ59I1n+wXcLDSNVs4EzgqPsOaU6kq0Qd+vurL++hKBYKLriNAtxkJpHB5NF8nbWQc8+DV12Pn+1f18MIx6dbfxmqhFpSIsZ2nbjtRUGc5WfmNT9i4E27VkvcnFx0MDuq1p8KtrQ7P4+MeNHzkqLE517SkWMRjP+44G5AppGX21tABSaZCen+KSDNLOjLqf7rHF+Oe3QJ+OAxFDBxJlaSjtHWgLxjReelNN9EpEtNO1MAOjgrHXrixN/wJr3YGU5O8EE5KPn+LF/tvCFnsCKY7bSdz+1Jydp92MqcbIEnu4ru5o14mgRMJRtc2Vi0gO+NGk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: alloc_contig_migrate_range has every information to be able to understand big contiguous allocation latency. For example, how many pages are migrated, how many times they were needed to unmap from page tables. This patch adds the trace event to collect the allocation statistics. In the field, it was quite useful to understand CMA allocation latency. Signed-off-by: Richard Chang --- include/trace/events/kmem.h | 39 +++++++++++++++++++++++++++++++++++++ mm/internal.h | 3 ++- mm/page_alloc.c | 30 +++++++++++++++++++++++----- mm/page_isolation.c | 2 +- 4 files changed, 67 insertions(+), 7 deletions(-) diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index 58688768ef0f..964704d76f9f 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -304,6 +304,45 @@ TRACE_EVENT(mm_page_alloc_extfrag, __entry->change_ownership) ); +TRACE_EVENT(mm_alloc_contig_migrate_range_info, + + TP_PROTO(unsigned long start, + unsigned long end, + int migratetype, + unsigned long nr_migrated, + unsigned long nr_reclaimed, + unsigned long nr_mapped), + + TP_ARGS(start, end, migratetype, + nr_migrated, nr_reclaimed, nr_mapped), + + TP_STRUCT__entry( + __field(unsigned long, start) + __field(unsigned long, end) + __field(int, migratetype) + __field(unsigned long, nr_migrated) + __field(unsigned long, nr_reclaimed) + __field(unsigned long, nr_mapped) + ), + + TP_fast_assign( + __entry->start = start; + __entry->end = end; + __entry->migratetype = migratetype; + __entry->nr_migrated = nr_migrated; + __entry->nr_reclaimed = nr_reclaimed; + __entry->nr_mapped = nr_mapped; + ), + + TP_printk("start=0x%lx end=0x%lx migratetype=%d nr_migrated=%lu nr_reclaimed=%lu nr_mapped=%lu", + __entry->start, + __entry->end, + __entry->migratetype, + __entry->nr_migrated, + __entry->nr_reclaimed, + __entry->nr_mapped) +); + /* * Required for uniquely and securely identifying mm in rss_stat tracepoint. */ diff --git a/mm/internal.h b/mm/internal.h index f309a010d50f..e114c647e278 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -537,7 +537,8 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long low_pfn, unsigned long end_pfn); int __alloc_contig_migrate_range(struct compact_control *cc, - unsigned long start, unsigned long end); + unsigned long start, unsigned long end, + int migratetype); /* Free whole pageblock and set its migration type to MIGRATE_CMA. */ void init_cma_reserved_pageblock(struct page *page); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 150d4f23b010..f840bc785afa 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6219,9 +6219,14 @@ static void alloc_contig_dump_pages(struct list_head *page_list) } } -/* [start, end) must belong to a single zone. */ +/* + * [start, end) must belong to a single zone. + * @migratetype: using migratetype to filter the type of migration in + * trace_mm_alloc_contig_migrate_range_info. + */ int __alloc_contig_migrate_range(struct compact_control *cc, - unsigned long start, unsigned long end) + unsigned long start, unsigned long end, + int migratetype) { /* This function is based on compact_zone() from compaction.c. */ unsigned int nr_reclaimed; @@ -6232,6 +6237,10 @@ int __alloc_contig_migrate_range(struct compact_control *cc, .nid = zone_to_nid(cc->zone), .gfp_mask = GFP_USER | __GFP_MOVABLE | __GFP_RETRY_MAYFAIL, }; + struct page *page; + unsigned long total_mapped = 0; + unsigned long total_migrated = 0; + unsigned long total_reclaimed = 0; lru_cache_disable(); @@ -6257,9 +6266,16 @@ int __alloc_contig_migrate_range(struct compact_control *cc, &cc->migratepages); cc->nr_migratepages -= nr_reclaimed; + total_reclaimed += nr_reclaimed; + list_for_each_entry(page, &cc->migratepages, lru) + total_mapped += page_mapcount(page); + ret = migrate_pages(&cc->migratepages, alloc_migration_target, NULL, (unsigned long)&mtc, cc->mode, MR_CONTIG_RANGE, NULL); + if (!ret) + total_migrated += cc->nr_migratepages; + /* * On -ENOMEM, migrate_pages() bails out right away. It is pointless * to retry again over this error, so do the same here. @@ -6273,9 +6289,13 @@ int __alloc_contig_migrate_range(struct compact_control *cc, if (!(cc->gfp_mask & __GFP_NOWARN) && ret == -EBUSY) alloc_contig_dump_pages(&cc->migratepages); putback_movable_pages(&cc->migratepages); - return ret; } - return 0; + + trace_mm_alloc_contig_migrate_range_info(start, end, migratetype, + total_migrated, + total_reclaimed, + total_mapped); + return (ret < 0) ? ret : 0; } /** @@ -6355,7 +6375,7 @@ int alloc_contig_range(unsigned long start, unsigned long end, * allocated. So, if we fall through be sure to clear ret so that * -EBUSY is not accidentally used or returned to caller. */ - ret = __alloc_contig_migrate_range(&cc, start, end); + ret = __alloc_contig_migrate_range(&cc, start, end, migratetype); if (ret && ret != -EBUSY) goto done; ret = 0; diff --git a/mm/page_isolation.c b/mm/page_isolation.c index cd0ea3668253..a5c8fa4c2a75 100644 --- a/mm/page_isolation.c +++ b/mm/page_isolation.c @@ -434,7 +434,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, } ret = __alloc_contig_migrate_range(&cc, head_pfn, - head_pfn + nr_pages); + head_pfn + nr_pages, page_mt); /* * restore the page's migratetype so that it can