From patchwork Wed Apr 3 03:55:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13614926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3610EC6FD1F for ; Wed, 3 Apr 2024 03:55:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F9256B0083; Tue, 2 Apr 2024 23:55:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A9A26B0085; Tue, 2 Apr 2024 23:55:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 670B56B0087; Tue, 2 Apr 2024 23:55:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 492106B0083 for ; Tue, 2 Apr 2024 23:55:24 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C54241C0E2E for ; Wed, 3 Apr 2024 03:55:23 +0000 (UTC) X-FDA: 81966855726.25.16CEC29 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf28.hostedemail.com (Postfix) with ESMTP id 1423FC000E for ; Wed, 3 Apr 2024 03:55:21 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=cZXm4Wxl; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712116522; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=i9BCiRwbqwf4wJug2EpE1JE+4HFe2zTO3BW9x4elXN8=; b=TxoDayLO044kKegZPzVi9k53/CH4ctw9P2+UMhgkZNfoop5woG595IJ/fi3uGV9mFYXSds 6SL26mspkntHHec9usnIOB+mcr6PZpOCR51/tW/V1WzxzuzHSTG5GCCyTxOizl5horWO5c u3gizztzAMHkNU79YXVIH0KZ8jg3FIE= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=cZXm4Wxl; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf28.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712116522; a=rsa-sha256; cv=none; b=eGZHgTJ7myKBZmm+0xk/BnjzHL+TkT1LhAf7QqPLo0u5qU40ct6g+fJJB3XfIFakICyGE9 DoI9aiiww61JB+sBIOvWKRloUBgHfe/AEER1St/gIY3iwgtgcVeujr7X1Qiwl7nX6ovZFl Eu8YZCZ0Z3fA4/VFjNo32YiFLkrzV+c= Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6ea9a605ca7so408600b3a.0 for ; Tue, 02 Apr 2024 20:55:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712116521; x=1712721321; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=i9BCiRwbqwf4wJug2EpE1JE+4HFe2zTO3BW9x4elXN8=; b=cZXm4WxlbNzv4KXNP2aHrArJj0qUT/ykAgXc0zKq6rT6xojC/3NXjJy1E03Obf/tcU M7xG6iBMmQ6RQV2DM6O0QwLkVwfUjPWISUPg+vRGH5PfWEPuYzvXJglHq8Y//lYxmiM9 JsoanzKvCqLM8IKg/A30aF7yzsN5GP+/uu1bcT/V3xtnzRBmiOjs7XXj+4/bRiSa2cyi HYGjEsCupAase6I5QW/KeVrViqvbYb4OFg8ianuU3lxYCmcHuKF/WsSO2Hqs2tgouq9I eL/khl4/taYhFnNDcW7Ytk9JD9yRVt5qq7/vweVHDwJcyGsFk7x7CU9bVRZDQoJQjeve J5gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712116521; x=1712721321; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=i9BCiRwbqwf4wJug2EpE1JE+4HFe2zTO3BW9x4elXN8=; b=roWkyzPdN89llOum3sTOsHPc4vs8Od8YtfN2BiNr6IZdYXSWkbRTrsIPioFX1h3kUX j/IO27lpJ8vReddb4dFOGibUFKr5cilThfkA4kIUWTPBDP5wVztFzMFVrYUhqHg9rWgb o+uEb2Din0lZxlKd7vA2XVbvSNDkz2c0MZdDD+Xdm9V5ySrGeTh31Y/oq1oD+UYwjhii t+2WRdGFqIRGvwek0jiiPBcOjI6X0TCm9HDuJx4cwULceF+jm5wM20r34/o+AV7MbecQ o32rGUv+4MEBy2Zj6kdLI6xMotBUXUI9Tg9BQDOqaIqRhh0rq+bIr+IhCdoCSk0ww2Kr dR2g== X-Forwarded-Encrypted: i=1; AJvYcCXeHFnx53kYT5Mu1pJL0HRkfnUdW99l0rW7BhlRb7FZC0fjNxQ+0yd3gVqH+Wp5hdnF4D16AoGI5ZKPGE47el/fVcg= X-Gm-Message-State: AOJu0YyX7OGJthL7se0NdzZHioajNXmbg/1i0UM9K446GOx201HF4RQw sLgHFK8HjQTYCfBiGZFOb0NWSp5JTc04w/KCwZrnz4v3JzbR2FMA X-Google-Smtp-Source: AGHT+IGSdjv7QFQzG6ublXb1ofzXODIa0dx8iqny+NY+6bOY3yXs2rgr8PPt1mIgDnYBlC+LWNL7Iw== X-Received: by 2002:a05:6a21:a58f:b0:1a7:242a:5aaf with SMTP id gd15-20020a056a21a58f00b001a7242a5aafmr1780981pzc.15.1712116520750; Tue, 02 Apr 2024 20:55:20 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r15-20020aa7988f000000b006e5808b472esm10922351pfl.95.2024.04.02.20.55.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 20:55:20 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: cerasuolodomenico@gmail.com, chrisl@kernel.org, david@redhat.com, kasong@tencent.com, peterx@redhat.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, yosryahmed@google.com, yuzhao@google.com Subject: [PATCH v3] mm: add per-order mTHP alloc_success and alloc_fail counters Date: Wed, 3 Apr 2024 16:55:02 +1300 Message-Id: <20240403035502.71356-1-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 1423FC000E X-Stat-Signature: zow3wkuj7i8qpum41udn9rx5beiijana X-HE-Tag: 1712116521-72678 X-HE-Meta: U2FsdGVkX18E1Uni2cB2241/0/0bIxXaq5TuAMmfxGmmXyb/XDOXL++3ce6uEm7RCmTYlkzkx/NnPpINzlQMwHO5jb+AOBOrmXm+1ErFWsHhLoLxz3cQgCsSA4YIiVRd0xiJy6kmkuYNyFlMOp1YoNgPLF7Ypj+2eaRt4SEQvr1eaMFy+mtdLgQoptcBaHtiGsGGR3OMOCAa71W/e4pJBJ94gleWR2H7KaXFrAIzXyai1Vz23a7ZQQ1d6UyW05KiKpyAnuUnX9PVI2xw41svFKt1+PpXKAKSQwxIYCleo3CcdP0H9W2vS0rHNk5rO0mzIZDpzVrwLp6aMw15bc5MOlp1Jy8sje64XF3qcoVeJuH4XB3eaRVWejDErZCSXLCe0JRgMFvM+Ciwhvax9Di8FC/zOTofj0hT0RD/Z6MHWb0qAnwjgwVNhQiOxs88lhazUqXRXsq3Tw1a4qNpulKEhsxfZIw66yxbSHuGVNjpJ9WzBTFt1AEkRQhy8Zp3sub1/71pH1jTf4hnYvMzau9asOghBBgzHPWGHrrjmvYIfD6C+W91SoCQM+yJ3e+M66hv00N0HP4n3eysGzdbFoWLwSClICo7CPypHM5pRJZaDrTFOX/7YPaJgxZdRRv8/3/+hg8Y9ZU3YgW/DtZwldDOnawahfuKtfgEmkMXdP2kw7XLqV/MgsPqbsIeeffhe5rRPcM96vi/hwa5Ybyhhfm+uzTw/IdJ3TH5g8aEYSAwHiYvoOchFUs0qSnKlcGezHg+qJ2GqVjcE+OOYOcSF+4fgstUew1ziPRK/oLiDJh+BLAgjktK94rNH+OicMBFlN5ETQsUPtpb3tfa7n6tWVMRowFCeZpYGNa68H1JWO6JUPoh/E0eU3oENk2HuZzK3rg4eP8Lz69B9wA2AsYXlSnd0UQYBauhVqGEyOtebAJd8/SvEhJNyrU90fqdsvczNDl9PtymoCn294RZhbh6JQB bdbmXLQw 481Bjn9UgkCzipW5ClDGjyvnQzdJSZTLNsbIxMHXW542FRBkjJJI53ZXuUGcL0StZNYGeTSoAf4vXbbZJghnovhwT4BQ+sj2jwxuqb+Ut+Avja76gZvRUIoFJGoSeZxn/MNzcnzrn5Aa53cj6QrvKF8yn1bqQcrcLeaU2FJHKZ/i2rouuieqgcfVw/BMnBAUuUhYKEF6uG+cZJpuP6QvsrnF9KxK3Ef6LdRSoSORQxnYgJSRsxC4aGva2YSPRk4QPCRxl8I6YnuiHN6tsEVRNEgNMMNK6/0SG0t9wGsfmSMpCMHZR206wI6SZRLUErxKqtQhYYkmp2VOIybF3Z18O9QiESJLh/D8FM+84HBbpgwAjbTnxyY3dR2P7vb6ZdPfKroyJLQ0wpz5XOZC5RpqUa2jW1w+7ri84C2E5mRel6QxRcxQIR6MyxkDgV+6VafPADN9TQmLDyJhfcZbGbgvI5PLMhRIfMHE6ebZqRrWPkGeXrVo5PDbnr/7BkcgB4VKOjyOntFrBPDbK31Z+rGX2ljEhB3sl/pIkvjG1ZKNHnqympWE71WjJJQsXlmJr62aHRPv4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need. Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory. This patch sets up the framework for per-order mTHP counters, starting with the introduction of anon_alloc_success and anon_alloc_fail counters. Incorporating additional counters should now be straightforward as well. Signed-off-by: Barry Song --- -v3: * save some memory as order-0 and order-1 can't be THP, Ryan; * rename to anon_alloc as right now we only support anon to address David's comment; * drop a redundant "else", Ryan include/linux/huge_mm.h | 18 ++++++++++++++ mm/huge_memory.c | 54 +++++++++++++++++++++++++++++++++++++++++ mm/memory.c | 2 ++ 3 files changed, 74 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index e896ca4760f6..5e9af6be9537 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -70,6 +70,7 @@ extern struct kobj_attribute shmem_enabled_attr; * (which is a limitation of the THP implementation). */ #define THP_ORDERS_ALL_ANON ((BIT(PMD_ORDER + 1) - 1) & ~(BIT(0) | BIT(1))) +#define THP_MIN_ORDER 2 /* * Mask of all large folio orders supported for file THP. @@ -264,6 +265,23 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, enforce_sysfs, orders); } +enum thp_event_item { + THP_ANON_ALLOC_SUCCESS, + THP_ANON_ALLOC_FAIL, + NR_THP_EVENT_ITEMS +}; + +struct thp_event_state { + unsigned long event[PMD_ORDER + 1 - THP_MIN_ORDER][NR_THP_EVENT_ITEMS]; +}; + +DECLARE_PER_CPU(struct thp_event_state, thp_event_states); + +static inline void count_thp_event(int order, enum thp_event_item item) +{ + this_cpu_inc(thp_event_states.event[order - THP_MIN_ORDER][item]); +} + #define transparent_hugepage_use_zero_page() \ (transparent_hugepage_flags & \ (1<event[order - THP_MIN_ORDER][item]; + } + + return sum; +} + +static ssize_t anon_alloc_success_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + int order = to_thpsize(kobj)->order; + + return sysfs_emit(buf, "%lu\n", sum_thp_events(order, THP_ANON_ALLOC_SUCCESS)); +} + +static ssize_t anon_alloc_fail_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + int order = to_thpsize(kobj)->order; + + return sysfs_emit(buf, "%lu\n", sum_thp_events(order, THP_ANON_ALLOC_FAIL)); +} + +static struct kobj_attribute anon_alloc_success_attr = __ATTR_RO(anon_alloc_success); +static struct kobj_attribute anon_alloc_fail_attr = __ATTR_RO(anon_alloc_fail); + +static struct attribute *stats_attrs[] = { + &anon_alloc_success_attr.attr, + &anon_alloc_fail_attr.attr, + NULL, +}; + +static struct attribute_group stats_attr_group = { + .name = "stats", + .attrs = stats_attrs, +}; + static struct thpsize *thpsize_create(int order, struct kobject *parent) { unsigned long size = (PAGE_SIZE << order) / SZ_1K; @@ -549,6 +595,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent) return ERR_PTR(ret); } + ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group); + if (ret) { + kobject_put(&thpsize->kobj); + return ERR_PTR(ret); + } + thpsize->order = order; return thpsize; } @@ -1050,8 +1102,10 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) folio = vma_alloc_folio(gfp, HPAGE_PMD_ORDER, vma, haddr, true); if (unlikely(!folio)) { count_vm_event(THP_FAULT_FALLBACK); + count_thp_event(HPAGE_PMD_ORDER, THP_ANON_ALLOC_FAIL); return VM_FAULT_FALLBACK; } + count_thp_event(HPAGE_PMD_ORDER, THP_ANON_ALLOC_SUCCESS); return __do_huge_pmd_anonymous_page(vmf, &folio->page, gfp); } diff --git a/mm/memory.c b/mm/memory.c index 912cd738ec03..40de2b7a73ab 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4374,8 +4374,10 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) } folio_throttle_swaprate(folio, gfp); clear_huge_page(&folio->page, vmf->address, 1 << order); + count_thp_event(order, THP_ANON_ALLOC_SUCCESS); return folio; } + count_thp_event(order, THP_ANON_ALLOC_FAIL); next: order = next_order(&orders, order); }