From patchwork Wed Mar 6 18:24:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13584429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F20EDC5475B for ; Wed, 6 Mar 2024 18:25:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C104E6B00A9; Wed, 6 Mar 2024 13:25:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B97856B00AA; Wed, 6 Mar 2024 13:25:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EBAF6B00AB; Wed, 6 Mar 2024 13:25:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8C0576B00A9 for ; Wed, 6 Mar 2024 13:25:28 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 70120140534 for ; Wed, 6 Mar 2024 18:25:28 +0000 (UTC) X-FDA: 81867441936.27.A882BD0 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf09.hostedemail.com (Postfix) with ESMTP id C8D8C140007 for ; Wed, 6 Mar 2024 18:25:26 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=v3XAv0Hw; spf=pass (imf09.hostedemail.com: domain of 3FbXoZQYKCFgIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3FbXoZQYKCFgIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709749526; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UytzFxvo2MN54m2UV3o6+O9sIke62l96NyrXCwye1Yc=; b=RMb8CEz/guOZ19phwxZhKVaZdaFG1VzbaVPnSmvDGI1B8qjoUsjgJgiOTWHEJg6WfM2msn hDrPk/TBQ+SBQzGRa1cChhlwtsACDhp1C0EWiC5A9Ct3P0KqM7PT/888k4l9OrJ+LIPxWP nXK046yt+vbNsZSG45MLmKwHKqSOK8s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709749526; a=rsa-sha256; cv=none; b=TFxOiqkUuGiuK6hzgCbtRqy85yi3bZ+zM/uRNChLOULGP/AkLPTyc4DMHucsZbi5VxUx8S 2yFtwEDPV6dNV1IIYYconyjqfQzXHdO9o/yrXShrWwELTe2oe5Db5fGRz1d3IUn6Db/E0n WRb8PWEGIoReM9nQuLksfyLXWIgoasM= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=v3XAv0Hw; spf=pass (imf09.hostedemail.com: domain of 3FbXoZQYKCFgIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3FbXoZQYKCFgIKH4D16EE6B4.2ECB8DKN-CCAL02A.EH6@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-60802b0afd2so837997b3.1 for ; Wed, 06 Mar 2024 10:25:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709749526; x=1710354326; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UytzFxvo2MN54m2UV3o6+O9sIke62l96NyrXCwye1Yc=; b=v3XAv0HwooKL09LfUVlYxFTeXst7s9zBnX3wEKoEO9NFUPiTiMKMuPj4sTON4TrSYf N9tKQ9Djr2ifsK3LoduK9rLcA9LZMCBxpHjuLowQO9SA3UGeCW29e5NSRTi++NAYoqFR Y+2lTksLcgLD30Dg5CKJF00RuNRh9djBHKbaBLMbE2B8ca0fB8aGfiHiq0i6f2tBLjCW HodMEQVlinqc4xJCQ0Cr4BAf1c78imIxvjV+5nmF+GgyL08qhYZnJM/RtGZDckwzFFGH CyjFY1up4Voy+e9DM/toD7K60ddvsLA+SdiOoi1x9yT+XJgGI3TJI9eYl5RSseJUDEh5 rxLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709749526; x=1710354326; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UytzFxvo2MN54m2UV3o6+O9sIke62l96NyrXCwye1Yc=; b=DOcy75TASD7SOwY2Y5XBDQLoosNHOchdWyuieC4AqZm4RowGSLBBdkvvzhB2yMmRPd UqFu40Ss9I0NYGbULqNvOLPSOe+eSQsbCmfOrjxBKngslnjdmnkOG3AFR9n6XI4YeB58 BzqrjPFsfKq0SHneL5urYdi9EiIYnqWCbOSz5Y0nS57kfcKD5baE5zn6lnkk/qM6qMVE wFB3jw3YWFxAReSGGynRPGCuFS3RJqxaGVnADktQlJtP+yYWaN3XkRwCjtuzP8Uz1O+F E8xL9D5R/y5j8wN+to8QhFTeaiW4/Iu2d5Ll46XBcfcTr+apO5b6qx9Epj3DfyrVauBJ kweg== X-Forwarded-Encrypted: i=1; AJvYcCWJz1claTCBvwrg5Nnls3BGPRr81dA0M0Wn6P9bcviB5XaW266suZPp6WwV6fq7pBRiZnRV7BwBjfmdtSfsvZoWtjU= X-Gm-Message-State: AOJu0Yx0aDSBnISBR6VFbEc5gHpw/nQrcOOqAnblHV1tQKs5A4I5Q8UM EODmWtHFbRuCupM+AbeADn5Ir2gbp0XZk6gFvfpJJEO1R6RnPPYO7Sw7G47iREDMnpccxaRfdFz mEw== X-Google-Smtp-Source: AGHT+IELFceSPWFrUjgphXkqF3QF81hXppl+wz2Z5M52XUCNjOCO0HkPXUz6VhR3ZLtCxZs2Az+yJ39E678= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:85f0:e3db:db05:85e2]) (user=surenb job=sendgmr) by 2002:a81:7948:0:b0:609:5bd8:de84 with SMTP id u69-20020a817948000000b006095bd8de84mr1217843ywc.0.1709749525732; Wed, 06 Mar 2024 10:25:25 -0800 (PST) Date: Wed, 6 Mar 2024 10:24:17 -0800 In-Reply-To: <20240306182440.2003814-1-surenb@google.com> Mime-Version: 1.0 References: <20240306182440.2003814-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240306182440.2003814-20-surenb@google.com> Subject: [PATCH v5 19/37] mm: create new codetag references during page splitting From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: C8D8C140007 X-Rspam-User: X-Stat-Signature: dz1ubf6ha9xnqht96wi7hepb6bmu5okx X-Rspamd-Server: rspam03 X-HE-Tag: 1709749526-122990 X-HE-Meta: U2FsdGVkX18R3rrcDgfapuFWC6OxCUMfXBIBvdKhJ4mxPeJt/wWPELtwVcnUBGs/lErWIVXTxi3fK/8I3K1A2p/o65TMDbhHpY7G6eDNFYG+GJ+Ja9kczT84FQLvrPMhRKQkkoOV1GU+S8VvaYPIPHw5HnTUNb4i+CDN8KpO3mMhJsMQIONihumDdGoxSkVf7ZVlRUV+x/dN/4rtqS3w7qeB9fh7Ozm/U1vrFoNSD6faR+2+UPllB17M877soFJVn7l+BneS2MQOzDLooGKKZGw3ayXJiBXszGmleFGKiXrZaj02R2zCWRfwhJv2xg0AOn4sT90HIdsk8Urpvt5IMB4GzcsaEqZyf/wgkmB6SgtiWp3s+AkHyo32UlYOnsP90J8MNBmJxdTCQ0qwNWlPzvx0FDySw1hJxpFMZv2o1yLGKXzAgx5qvk7hFGdEx0pK++BMFY2qVa4Y9ashM7aKIi6Amc6z2CUx97DP9S2jT5DithOtyFLjQC4zoUGiPSE5Wfv2MzNDa83LX4YfKi3xFg34v8MxE7Atko9tyCjezfkXFVRrXtxIfsUukvE7NPz8u/nbT/D2THVgcG0aGAXyUt6ue//9OH4UYv0qoNwy9EfzMB9nBYcUdUcS6BOgGnyjmeP0s7Hzs+9dz0P8wP2FARg8TKiyJImweZlNXvIWIsrplaXYZZbMgqfsYjYleEP29TFbE4d3hb2TAX6t3BPsflQ8Zw9J65GCaLKMORuDQGWYsKLIQuIVWvk9qdDP6bSgDjfm75bD0sFmAxjFcyENDIXrAtWIoX12mj5c7D/gPwh1M1h/5uNMrqzJjKJEK+xXUo6qwvTiiS59DvcXNadftII8+UtQ2ctsH4VPLf10czAsxpnAQpe2ubP7JjnG9m8x7aSM2hiiqOFylyrInCNK8BC21xHd+KJhYicYRoU2SPD2Ca7I8Qt4HJsOqB2FpWSICAcZUSgVRnPlo6G9AfR WS2QRnUp A86AqB6YHxFiYqrMEt5NdRPbaF6DfBIVb75dZ3V7hKFwGpIYZjTjDF7k+1SJ8vybuRngOB7AtnCbxkDdtr52a6WkqyGoqT2EVQG5jS+7X61g7YRhwyUjdxaPcVccNr1bZPg/1bYROP6n7/M8lhXRVWBCC3qB9rcrbyrFsmMfmPoSBQIfd1UFv7TJMGuOAM+4eOStf87KqaQdvJpJY4rt2SADi3OtsyH6I32xSxN1KfEKheml7QuUyUZzSvpcdTySKVzp9SWgiOSOzpqRSel6tx3BF3YkDaCFY/x/hbeT9C+Ifb7BAkZi0p+eEP8P9Bx2ui+dqfeIicuskzSbI4V5Gqki7rtmgCGmgavuYqtDq7fRQUkmhOcvFo5mW43OXSXkqDqe7GLKVkYiEKE7XIWlvymrPQYr9yLLGGjUD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a high-order page is split into smaller ones, each newly split page should get its codetag. After the split each split page will be referencing the original codetag. The codetag's "bytes" counter remains the same because the amount of allocated memory has not changed, however the "calls" counter gets increased to keep the counter correct when these individual pages get freed. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/alloc_tag.h | 9 +++++++++ include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 4 files changed, 43 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 28c0005edae1..bc9b1b99a55b 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -106,6 +106,15 @@ static inline void __alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag this_cpu_inc(tag->counters->calls); } +static inline void alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) +{ + alloc_tag_add_check(ref, tag); + if (!ref || !tag) + return; + + __alloc_tag_ref_set(ref, tag); +} + static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) { alloc_tag_add_check(ref, tag); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index b49ab955300f..9e6ad8e0e4aa 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -67,11 +67,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) +{ + int i; + struct page_ext *page_ext; + union codetag_ref *ref; + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + page_ext = page_ext_get(page); + if (unlikely(!page_ext)) + return; + + ref = codetag_ref_from_page_ext(page_ext); + if (!ref->ct) + goto out; + + tag = ct_to_alloc_tag(ref->ct); + page_ext = page_ext_next(page_ext); + for (i = 1; i < nr; i++) { + /* Set new reference to point to the original tag */ + alloc_tag_ref_set(codetag_ref_from_page_ext(page_ext), tag); + page_ext = page_ext_next(page_ext); + } +out: + page_ext_put(page_ext); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int order) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int order) {} +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a81a09236c16..d596449b5bc8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -2946,6 +2947,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, order, new_order); + pgalloc_tag_split(head, 1 << order); /* See comment in __split_huge_page_tail() */ if (folio_test_anon(folio)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index eb5cae9b967d..39dc4dcf14f5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2663,6 +2663,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, order, 0); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, order, 0); } EXPORT_SYMBOL_GPL(split_page); @@ -4850,6 +4851,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, order, 0); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, order, 0); while (page < --last) set_page_refcounted(last);