From patchwork Fri Apr 12 11:48:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13627657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A51E0C4345F for ; Fri, 12 Apr 2024 11:49:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3444C6B0089; Fri, 12 Apr 2024 07:49:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F4366B008A; Fri, 12 Apr 2024 07:49:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1953F6B008C; Fri, 12 Apr 2024 07:49:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EB13D6B0089 for ; Fri, 12 Apr 2024 07:49:21 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A775A80EBF for ; Fri, 12 Apr 2024 11:49:21 +0000 (UTC) X-FDA: 82000709322.12.B6B65BD Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf13.hostedemail.com (Postfix) with ESMTP id CD8DD20003 for ; Fri, 12 Apr 2024 11:49:18 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ww5Nz04O; spf=pass (imf13.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712922558; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EBqcW3yGOLnf8fCkLXd+q0+NdDQ6JnFQ/vzKSqccdZY=; b=iq26dWwpTwaB02sXTuCAlYsx/I+XmnJ9X4N0BZblZ172pJpKoPn6FxEMkKzM3lUK0t/kZF mfA/9ptNA326cx0Lg0bkafTA7MrZqKPh8b24zmGUaflMJd0lFg6gSLYYVYmDaW4QpGiFo1 W2GRbp79S97ZFszHEm9kukjJO2ABY9w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712922558; a=rsa-sha256; cv=none; b=acGbzv54QqFInP6Yy6cdGmpAxebF6xsYjX217p8M5eGFQG5X5cQZZz52sa9Co0aqo+Nu3Z cEm1SrFH4x3HpbvAZJ1rC3dUnhtp5S4VD6ePqA24HoLGPOkDiRmuYNhog+GlM29WQHKuYB +SUxL86zOXMMAmgNA9FXd1FflHFetOo= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=Ww5Nz04O; spf=pass (imf13.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6ee12766586so553239b3a.0 for ; Fri, 12 Apr 2024 04:49:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712922558; x=1713527358; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EBqcW3yGOLnf8fCkLXd+q0+NdDQ6JnFQ/vzKSqccdZY=; b=Ww5Nz04O1IvaAyYmgMH7B/JGv8ruDgaog3SgDUwbIhsCrq2nZwSTxpGcUBLq9xETLG M6Yzo7YshZivCGxzbokfjxdrqj729AFvH9GOrbjmnpE6+xhUiZM7W22uY+I+WLdi8faI FKxaLs7TU4teAjQJPnNU0PHg2WQm2VNSOHCi6rjn6xEQnDvJEPE+1qgI4MRB1CbGKLju LdbIrpXIrQsOHswnsyki2sc5PSEto8FVvlzG7o0RtCWD/VKawtOyEbYOb/Dy6GEcMJpq U7qQ2LaGZzGYZF8GL0HF9BptjiIS75MKzUThKEwSgBjbU4qLPXMKArR4LNqW0iCDOpVI Tqvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712922558; x=1713527358; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EBqcW3yGOLnf8fCkLXd+q0+NdDQ6JnFQ/vzKSqccdZY=; b=UiwA+mNXXpuSFfNuOTjV1gQpNIq0Ji8M7sAMY0rpVlXjBJCYPhOqDQCB8ZpkyUY3OA ca1e51pkOHkH6ylnobm+gLIDP1FMHgnXBuVxfPPwq43MXxzuHTNQj1sVBtVk75XoTACI q9lLHVI04dnQDNf32ZEbdIlvry/QRDWrEmt5lqBiOP8AMTJvolChSJhk6TSoiPkPE6WE 43yRpecEGRVybUkZRSRGm5iZ88L0pq8UdpEK8qrKwqUNv/xP6snJl+JzH9WDKkmmR22q PW3F8TTLSHUS/ZVKwZU9vf3uTQLqqpZUQtCDfB5mMHYVobM48DlIg80IatwJi5mmrDES K8QQ== X-Forwarded-Encrypted: i=1; AJvYcCUf02gRmBUVfRWovMuZx+5vFYeCJTxlupyQXVOhz0uwz4En4cuf/TEQX/+qrQU7z2wl5fX37KU3Sl7yGC54UzDLDr8= X-Gm-Message-State: AOJu0YzInFDPvIlpdSDyePbyGmIJl4ZRx2nVqdiHakSgAKJGMuAI0NB3 rsay8vf3/+OSy3O1U4nCFrznTqs+NIwSeIqVMjc89WBUF8zuE2DN X-Google-Smtp-Source: AGHT+IGaTHWz4rwtGHc+sgkHopCT3nCaKwwOngZTn+zC3VlvQfYHhe+0iaWXVFi2u44WZCge+8Zqcg== X-Received: by 2002:a05:6a21:3406:b0:1a8:58a2:ea41 with SMTP id yn6-20020a056a21340600b001a858a2ea41mr3325699pzb.9.1712922557669; Fri, 12 Apr 2024 04:49:17 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id q1-20020a634301000000b005e43cce33f8sm2541048pga.88.2024.04.12.04.49.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Apr 2024 04:49:17 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: cerasuolodomenico@gmail.com, chrisl@kernel.org, david@redhat.com, kasong@tencent.com, linux-kernel@vger.kernel.org, peterx@redhat.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, yosryahmed@google.com, yuzhao@google.com, corbet@lwn.net Subject: [PATCH v6 1/4] mm: add per-order mTHP anon_fault_alloc and anon_fault_fallback counters Date: Fri, 12 Apr 2024 23:48:55 +1200 Message-Id: <20240412114858.407208-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240412114858.407208-1-21cnbao@gmail.com> References: <20240412114858.407208-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CD8DD20003 X-Rspam-User: X-Stat-Signature: udodp6mw7yox9bxq843e3kh9t5niyauc X-Rspamd-Server: rspam03 X-HE-Tag: 1712922558-375419 X-HE-Meta: U2FsdGVkX1+ZYsToizWr+KKRiGAOvng6m21BHg9k1GxftN9Wv/ZFCTfgmwUgd78TziXgiw5azt3hPdVKjEAU1RygeXkmcJSdrX63VvGwCEPvhV0ilwUC3SCUP2w9gSCVT9DgMFqUAxh42M6ifE9V0oHcceR8+XNSwq7Cav7VdwYA39tLgJDtuQCCr+CUrS1nkeu46xyGe+rNEUGZcpLixsiyYTKRbTrD5udise1jIvt2IDXaxwarVzrJg/v3BtTuKoVqe6NVWqqvBIqeCuHRLoRwR4lBPAW6nzeLpN5wqrit44dYEdPf5adx1zWaHFOmtxPVvZ8ARkoQCD1GLgzibaCoVjLY3o8b5jq5hXxoSymK8MJpqfLgsg3MI72cqCnrJOVqo3rq5RbQyGc36Lvy0GYp0gvsY9dJK+PivoQebZNt7WDc07iMcWnJDoOU0bi1Yy43fzGfpdflW2MEiG6Qyc7UC5ubkZyQTzjsKf2X24k3o14NmOspTgdZCAsqHleypBNja1cd2YgEM36e7jQKAfrYoWzm3GI2NSUbbX4k+bLqh4GfRUzCBCipBP81lUuuDGDa67uZu83ekoavLh/wDcr5xDLACrRZeRAx24H0lhMVOcbXbv2kz7whfS5ie+caDCbI9bwNdQVD0Jmo7A6aCBm6KFYFm8iH5QSeQk6AMGnBTvJA3kAHoYe3dtQbfeSu7+on3/TsFJB/vdAU4DDt9eK0n/uHgooLEFEWCAUq2YRRGnULsqhyqt/NlKs8OYNjlErGvEH6q7Dklvwyz9Vl7+tDyi1zliVSUPnhNNAqxACZ7+UrGSLdhXjuOUcowrBpT4oUEOZwpCXsNKYlo1DPBn+zWVfbADDxdFQb76v5oTWH8roMQoCxGxCc6mOd/oo5UDkbp9dTOlxYI5Jdh5r4L+YL5+9AWTThIDarG7kl8By+uOwHO34OWWLVuDEgm2HJEJ3/6rYeJOsLA+Q3VR9 UC0dGBtT eoa2rxIop5pJlrVQS5kPwNve7nU/rTpOldGP8qKFmJT7vVEU6VZ8AfigpiXkSKA8zD+hDQU0opXA3SuUzSHPtEXhGILMskZn4OpWQ7+Jpqt8AEQAliaT8f77IHgtlUUBu1nsYrbx3esb0pzkDZjFEjAVSmHHmnwqpbJVQfK+Ql61RowtjvsqZ2WfguCR5qNkn2QUgMCuhmdK603eJJwZWT/9M3XN4eUrBCnbeIc3ehNWAXscbzeMCjp2XTfsBXzk0bGe5Co89yF5Fa6USvJSR0tdHiyeFtVxmS3py5J0yHAvTIyi9r+4EE9sugtRX/qp6BlUMBNkkZWYtO9wA/yjYmn8iRTeiX9x7nQYG4bOeH/29CTG2TRhhFGzXEaBquQxcWyZjQki0DDEL00bfSaVOWntr4oJ43wvm+W6oY/JmmGUUChtNt8oXsfImk2Z6nsG52e9nsc2suu3KVw6kuAwDiIYQbI+qNSJSjMX6NTqrpIjdYwduyCsh0yqPV1OXzXZwq2aWsXjNI+V9uuWxzCC4XPM/CUX3bDd7ryr376LXkVP1XQ+zrR2QBDiq76jIh6R27rIvxGTCLo+Yb1yBFhcNYzZKv0JyymjpXbxoumPDDeCf5dQp9hNgiHNeU41UZLarOmvt8oB2yrnGTTKKQJJaMvCx0I/ECT0Zh5CDFfSSSTHxej91mrgsGBH0rsSXGp2iK7NKfo+L88Dl1brH1dMqZUNsfImVSw+BaX37fKHn9Wk0ErFcuhcILrH+McvDF9DMduS211F5fpZuxi5kmGG8+rKbSQWc9i9GOq+6srHkkxPtdAAdQM3nXeRjbaFjYZQTOC+KT05DNltCDRy9o19qGcgBEA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Profiling a system blindly with mTHP has become challenging due to the lack of visibility into its operations. Presenting the success rate of mTHP allocations appears to be pressing need. Recently, I've been experiencing significant difficulty debugging performance improvements and regressions without these figures. It's crucial for us to understand the true effectiveness of mTHP in real-world scenarios, especially in systems with fragmented memory. This patch establishes the framework for per-order mTHP counters. It begins by introducing the anon_fault_alloc and anon_fault_fallback counters. Additionally, to maintain consistency with thp_fault_fallback_charge in /proc/vmstat, this patch also tracks anon_fault_fallback_charge when mem_cgroup_charge fails for mTHP. Incorporating additional counters should now be straightforward as well. Signed-off-by: Barry Song Cc: Chris Li Cc: David Hildenbrand Cc: Domenico Cerasuolo Cc: Kairui Song Cc: Matthew Wilcox (Oracle) Cc: Peter Xu Cc: Ryan Roberts Cc: Suren Baghdasaryan Cc: Yosry Ahmed Cc: Yu Zhao Reviewed-by: Ryan Roberts --- include/linux/huge_mm.h | 21 +++++++++++++++++ mm/huge_memory.c | 52 +++++++++++++++++++++++++++++++++++++++++ mm/memory.c | 5 ++++ 3 files changed, 78 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index e896ca4760f6..d4fdb2641070 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -264,6 +264,27 @@ unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma, enforce_sysfs, orders); } +enum mthp_stat_item { + MTHP_STAT_ANON_FAULT_ALLOC, + MTHP_STAT_ANON_FAULT_FALLBACK, + MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, + __MTHP_STAT_COUNT +}; + +struct mthp_stat { + unsigned long stats[ilog2(MAX_PTRS_PER_PTE) + 1][__MTHP_STAT_COUNT]; +}; + +DECLARE_PER_CPU(struct mthp_stat, mthp_stats); + +static inline void count_mthp_stat(int order, enum mthp_stat_item item) +{ + if (order <= 0 || order > PMD_ORDER) + return; + + this_cpu_inc(mthp_stats.stats[order][item]); +} + #define transparent_hugepage_use_zero_page() \ (transparent_hugepage_flags & \ (1<stats[order][item]; + } + + return sum; +} + +#define DEFINE_MTHP_STAT_ATTR(_name, _index) \ +static ssize_t _name##_show(struct kobject *kobj, \ + struct kobj_attribute *attr, char *buf) \ +{ \ + int order = to_thpsize(kobj)->order; \ + \ + return sysfs_emit(buf, "%lu\n", sum_mthp_stat(order, _index)); \ +} \ +static struct kobj_attribute _name##_attr = __ATTR_RO(_name) + +DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); +DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); + +static struct attribute *stats_attrs[] = { + &anon_fault_alloc_attr.attr, + &anon_fault_fallback_attr.attr, + &anon_fault_fallback_charge_attr.attr, + NULL, +}; + +static struct attribute_group stats_attr_group = { + .name = "stats", + .attrs = stats_attrs, +}; + static struct thpsize *thpsize_create(int order, struct kobject *parent) { unsigned long size = (PAGE_SIZE << order) / SZ_1K; @@ -549,6 +591,12 @@ static struct thpsize *thpsize_create(int order, struct kobject *parent) return ERR_PTR(ret); } + ret = sysfs_create_group(&thpsize->kobj, &stats_attr_group); + if (ret) { + kobject_put(&thpsize->kobj); + return ERR_PTR(ret); + } + thpsize->order = order; return thpsize; } @@ -880,6 +928,8 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, folio_put(folio); count_vm_event(THP_FAULT_FALLBACK); count_vm_event(THP_FAULT_FALLBACK_CHARGE); + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_FALLBACK); + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); return VM_FAULT_FALLBACK; } folio_throttle_swaprate(folio, gfp); @@ -929,6 +979,7 @@ static vm_fault_t __do_huge_pmd_anonymous_page(struct vm_fault *vmf, mm_inc_nr_ptes(vma->vm_mm); spin_unlock(vmf->ptl); count_vm_event(THP_FAULT_ALLOC); + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_ALLOC); count_memcg_event_mm(vma->vm_mm, THP_FAULT_ALLOC); } @@ -1050,6 +1101,7 @@ vm_fault_t do_huge_pmd_anonymous_page(struct vm_fault *vmf) folio = vma_alloc_folio(gfp, HPAGE_PMD_ORDER, vma, haddr, true); if (unlikely(!folio)) { count_vm_event(THP_FAULT_FALLBACK); + count_mthp_stat(HPAGE_PMD_ORDER, MTHP_STAT_ANON_FAULT_FALLBACK); return VM_FAULT_FALLBACK; } return __do_huge_pmd_anonymous_page(vmf, &folio->page, gfp); diff --git a/mm/memory.c b/mm/memory.c index 649a547fe8e3..f31da2de19c6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4368,6 +4368,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) folio = vma_alloc_folio(gfp, order, vma, addr, true); if (folio) { if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) { + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); folio_put(folio); goto next; } @@ -4376,6 +4377,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) return folio; } next: + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK); order = next_order(&orders, order); } @@ -4485,6 +4487,9 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) folio_ref_add(folio, nr_pages - 1); add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_FAULT_ALLOC); +#endif folio_add_new_anon_rmap(folio, vma, addr); folio_add_lru_vma(folio, vma); setpte: