From patchwork Thu Mar 21 16:36:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13599176 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27939CD11C2 for ; Thu, 21 Mar 2024 16:38:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA0696B00B0; Thu, 21 Mar 2024 12:37:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4F616B00B1; Thu, 21 Mar 2024 12:37:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A51EB6B00B2; Thu, 21 Mar 2024 12:37:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8A7636B00B0 for ; Thu, 21 Mar 2024 12:37:54 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 387BBC01ED for ; Thu, 21 Mar 2024 16:37:54 +0000 (UTC) X-FDA: 81921602868.17.4492CD5 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 92A3518000F for ; Thu, 21 Mar 2024 16:37:52 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ErEUPVvl; spf=pass (imf16.hostedemail.com: domain of 3X2L8ZQYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3X2L8ZQYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711039072; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VNGm+8y6Ss+mlVcFwBeCqCAgvw3+NOVbq/tJOyMa3ww=; b=4/15QAj2hK73Uu7SsNt+IHQ1eWYjJPcu6vQ1Z9T79db+C17bL2Vkrj0MvN6+IXr9lLae5p 4/ptjMd462wyxX5F1ha+xyDW+vWTU7T07Qnsc+HO7J/wbezI9/0cWMnywKpiULg3rDsrcp KnVB4fFnGqM54itqPmznu+4TlDX303M= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711039072; a=rsa-sha256; cv=none; b=GFsgv1MOjKA1+kv9lh3m0Ue3AvbcO/aUKvfT0urgVjrAokNRzl1XPh6nkX/W7mXFovKFtx 9qGcxR70w+TO2wkELcXLfj7pvRma6zM/6aohmegl1Czn9Ls/sjevwRh3U3V8K9Lr/feMc1 lmwmuWgaCx1bLZD9X/KhFVAKt9NZxcg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ErEUPVvl; spf=pass (imf16.hostedemail.com: domain of 3X2L8ZQYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3X2L8ZQYKCEw685s1pu22uzs.q20zw18B-00y9oqy.25u@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcbee93a3e1so1860765276.3 for ; Thu, 21 Mar 2024 09:37:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711039071; x=1711643871; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=VNGm+8y6Ss+mlVcFwBeCqCAgvw3+NOVbq/tJOyMa3ww=; b=ErEUPVvl3oLXhsZkpm8au07HJNeO9uBhEK6OuDKX66gMbfrGvO0OmDAgAdaLfbnK+x 0Ed1nP8xBAcmpuries1V2ePle7QN2DWZR6mQoaEbFTLX/8GjKaqhftspQ2pxbLVFMGZ9 1QFm35+QKP/8ShIBYhPEbXJICUULU2+BCvtkmmc1+jamtG3VP/GjSXiF4QHyU3DDIt1+ lNQBY4sKe5zi/YSRpZuV2xOBDaqfMji8ashAj9kDNkYxapYJOZ0qJdNx+EbE97j6z5cB UnGH+bS4EF4sUmN/RMpjNU7TdHvjgcQmyrfVq28GKbaXpYC3HL8GWndrgzAuBlxqZF7T g24g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711039071; x=1711643871; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=VNGm+8y6Ss+mlVcFwBeCqCAgvw3+NOVbq/tJOyMa3ww=; b=WCwYGGJfaojcmk8mINMQI+KOJ4StxSPMDHERmfh1AIA1PTP1pwao8oU6yL2WThMlwE +i9F4od+BvYiDrlQEHz3lZ5jwxD4iBFB4XJ7LF9Wotmwut5ydR51ACuumk3D7PIzX7BC iovWaO/XFFK2bgchtrVx7YSICS3jYb9q3s8vpmbg0BPs8ZlBHTd6svq/piskf2R8Wx5W +Fzn2h5/BH0r/9NhPR2hKU/EqEv/WKYbwfInwg87jgx4IiKRlAljlxrP5oyT0HXPGOFb KIDxNzhc/vfUTiUFjXYmhnEyTpko3c364Nz+2mp/3jH1JdC1/k0gQVOod8O966mrkSRY 0bpg== X-Forwarded-Encrypted: i=1; AJvYcCXkRxxgeDTdzztM1vrACjNIu4K/pH9BtAjgypFPolPb9hSih5iDTV/PdWuhqaBTId9kpi0hbd2lISYi9uPD4xV9EKg= X-Gm-Message-State: AOJu0YySUEDZp2gfmxcFgubw479R87T1OdyQjiVlpd2SZ4d3Da1on4jH 2Mt3HxFESfcJSZrIaMO2nEs1KwP+NGi+ZH5dO/9jvkC7PEOSmYktnps92G3OpQKfRvadt8afK/P zSA== X-Google-Smtp-Source: AGHT+IFHuXgNI5SVUJASc8Fg628bHOvpHsCOaQtXol/znxQ1foKXK9QPyvtw6ludSGd9C7o69Hlt2+8i3RA= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:a489:6433:be5d:e639]) (user=surenb job=sendgmr) by 2002:a05:6902:1004:b0:dc6:44d4:bee0 with SMTP id w4-20020a056902100400b00dc644d4bee0mr1149298ybt.7.1711039071427; Thu, 21 Mar 2024 09:37:51 -0700 (PDT) Date: Thu, 21 Mar 2024 09:36:41 -0700 In-Reply-To: <20240321163705.3067592-1-surenb@google.com> Mime-Version: 1.0 References: <20240321163705.3067592-1-surenb@google.com> X-Mailer: git-send-email 2.44.0.291.gc1ea87d7ee-goog Message-ID: <20240321163705.3067592-20-surenb@google.com> Subject: [PATCH v6 19/37] mm: create new codetag references during page splitting From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, penguin-kernel@i-love.sakura.ne.jp, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, jhubbard@nvidia.com, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, vvvvvv@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, songmuchun@bytedance.com, jbaron@akamai.com, aliceryhl@google.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Queue-Id: 92A3518000F X-Rspam-User: X-Stat-Signature: fwnfweiojia5s7ygjj6rjnbxg93m7yqt X-Rspamd-Server: rspam03 X-HE-Tag: 1711039072-691852 X-HE-Meta: U2FsdGVkX18HhLita1f2z1FwE+QH/ittQGZpF5cnYVqkAQSR1v/Rw+myoJqKMCQtYO3UT7btr8ozEBazmdgOTOS/scadB1OUb4XoBt1iBOIdSaLkPApDF1jzIYNt8sScCs2O/5y5fmlGoREQhUla1lq0hdamvQqu31NpVwFEfBe7koFx6t4Zr3BSnVbYWpMgSxBKxK+TyZF80pmLIKE2VoTBiohSxci6jlRGXE+uK/PvecsfKourwIj/J0+bdLasKnjuhjCvqK7FIvdd5lWHpSdl+OGMykrTWV72/pSY6dtH+3SmrOPOyfCySuPRtqevEbmD7FO+dWE/j3s7OqzrRGUZD2o0NKeyniw1hDvr4Kn+dtjVD/dEnAykusZfHZFDYMPGQlU0ZDtX1cksSQ2G/YZGJXcD4DHVmH+7Qr0MwsW+0IfQdfugjjdzKhvPlBhDe4a/kxKs4RK8Yk0+oxwh76VJzb9fIFhUTllGc1lNC7fz7n4J30Hs2e3LhPhuTAfGitO9N/QMPwL2wJfRhRjfNeC9jR3TapC7xS9X1A3cPPKsmsrtdyBwfVi25+L2iW7BcFwyN2FZv9Oz5trF0UTob+UUdkCfwG9A+u3e53ZhuWiKEdqFX3Foz7xTdboNFUMmnrdBhxfx2m/WL0jqVlqfboyRrvW9uGNKKQdO4Khs4TP99DMUlgUiOT4tXiD7By2yJBYoMgUi+V2H7nvEmKALCR/6A1tQa0TMfn1OBoGzNQx0igxszURjFJKYxZk3FefXz9bbVQjsqIVL6aE0hyH+m+80tXA5bsWqQO7MlZxOu5UcfO3eV8ratLMkun1PBxzNvGQosICYOARgPBHKTInZ+67yXAm/+JIK0Md6pZpxuzzUoeGU4GHX4L2ShafojDFFw8gYJfZljJzzV5RP4YqTU8aDghqA0m+AhC/zDOv7J98TigQ2BnygMJ+4hYe9kUdEFotAlgEpBkBDyTCe1vX 7rjn2/C7 IdER9RG3g8ZTmudJ5j4fqpnnqKxyeggd57ef9w+DEQ/Zx0z9GKAzW91sXcPQ7wbnpYu3vaslCcBvtDUsY1AeYeDOdeYtYkbzC6PS3abtIjjVtWhVDA36ohGJtYKnTpdCmYYxOsBl5WhsRrgkHsRAau8+Gn6smLGudxFxYT+0XjfLQuJIy42ZwFJeCLYJzc6wkA/mNJwAy56/KHV0Pt7WR5K8yI5h7YIcu1uiqNIBqrAVqTprZIeJwPzBzXNalnpenIJl42DUvYogwHYyoOyEjT76Q+Zqb8CXIsqUfGYTTtbax8Ke5Qa70wNCw9A5nAzwQJDr37jPumN4J0dlKpufiXlwvzns3Z2UKXTyRU491ETmZnkRYexnXBO5v9JEzc1Gt5fUFZvQYZ+HhZ+mPl1s8ss9y30sMtusyWHfAS5hC9Ausl7A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When a high-order page is split into smaller ones, each newly split page should get its codetag. After the split each split page will be referencing the original codetag. The codetag's "bytes" counter remains the same because the amount of allocated memory has not changed, however the "calls" counter gets increased to keep the counter correct when these individual pages get freed. Signed-off-by: Suren Baghdasaryan Reviewed-by: Vlastimil Babka --- include/linux/alloc_tag.h | 9 +++++++++ include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 4 files changed, 43 insertions(+) diff --git a/include/linux/alloc_tag.h b/include/linux/alloc_tag.h index 28c0005edae1..bc9b1b99a55b 100644 --- a/include/linux/alloc_tag.h +++ b/include/linux/alloc_tag.h @@ -106,6 +106,15 @@ static inline void __alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag this_cpu_inc(tag->counters->calls); } +static inline void alloc_tag_ref_set(union codetag_ref *ref, struct alloc_tag *tag) +{ + alloc_tag_add_check(ref, tag); + if (!ref || !tag) + return; + + __alloc_tag_ref_set(ref, tag); +} + static inline void alloc_tag_add(union codetag_ref *ref, struct alloc_tag *tag, size_t bytes) { alloc_tag_add_check(ref, tag); diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 66bd021eb46e..093edf98c3d7 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -67,11 +67,41 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) } } +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) +{ + int i; + struct page_ext *page_ext; + union codetag_ref *ref; + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + page_ext = page_ext_get(page); + if (unlikely(!page_ext)) + return; + + ref = codetag_ref_from_page_ext(page_ext); + if (!ref->ct) + goto out; + + tag = ct_to_alloc_tag(ref->ct); + page_ext = page_ext_next(page_ext); + for (i = 1; i < nr; i++) { + /* Set new reference to point to the original tag */ + alloc_tag_ref_set(codetag_ref_from_page_ext(page_ext), tag); + page_ext = page_ext_next(page_ext); + } +out: + page_ext_put(page_ext); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline void pgalloc_tag_add(struct page *page, struct task_struct *task, unsigned int nr) {} static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c77cedf45f3a..b29f9ef0fcb2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -38,6 +38,7 @@ #include #include #include +#include #include #include @@ -2924,6 +2925,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, order, new_order); + pgalloc_tag_split(head, 1 << order); /* See comment in __split_huge_page_tail() */ if (folio_test_anon(folio)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 9c86ef2a0296..fd1cc5b80a56 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2666,6 +2666,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, order, 0); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, order, 0); } EXPORT_SYMBOL_GPL(split_page); @@ -4863,6 +4864,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, order, 0); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, order, 0); while (page < --last) set_page_refcounted(last);