From patchwork Mon May 1 16:54:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13227657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 585F0C77B61 for ; Mon, 1 May 2023 16:56:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94C2990001A; Mon, 1 May 2023 12:56:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8D664900002; Mon, 1 May 2023 12:56:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 79DA490001A; Mon, 1 May 2023 12:56:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 67474900002 for ; Mon, 1 May 2023 12:56:01 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3B42A12075F for ; Mon, 1 May 2023 16:56:01 +0000 (UTC) X-FDA: 80742288522.10.33889CE Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf07.hostedemail.com (Postfix) with ESMTP id 5A22C40022 for ; Mon, 1 May 2023 16:55:59 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="7gQ7i/dA"; spf=pass (imf07.hostedemail.com: domain of 3Hu9PZAYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Hu9PZAYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682960159; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+1Ptlsu3HrvWcynZoOK0ZqIIaOTUmOtwKZY5J1VaLvs=; b=NKMPrie4R+WoY2NrgUfJOVL9zYAgZVHbtjCi+eVSZcCoFK02A74gVBn6eqYowSyH73WU95 7sWreKniWnjIgcOWgiTLjolX9sq23yKpRqwsp4wOxQArdhbsIV8DTN8uG7YYZgYjHfVB85 PaOkeyHHS/nAZ6izB5hbTT681jR5iAE= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="7gQ7i/dA"; spf=pass (imf07.hostedemail.com: domain of 3Hu9PZAYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3Hu9PZAYKCGUVXUHQEJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--surenb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682960159; a=rsa-sha256; cv=none; b=V3D2qkeirT5Eyv0ILkDl7o9iHYanRGq3nw38JaJ/jTZHGncqsDFUmjC9aJYr+ON+sBrM0Z 5mnD138zzLczJMxrhxivEd09+lVddoqDhbSWT0DBM6Bti+0/dlDo7aWKQM65PrF/fDhKtf IFL1TKouSOiKfv/qmz0ZZ+3kS//q28A= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-b8f324b3ef8so3225150276.0 for ; Mon, 01 May 2023 09:55:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1682960158; x=1685552158; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=+1Ptlsu3HrvWcynZoOK0ZqIIaOTUmOtwKZY5J1VaLvs=; b=7gQ7i/dAC1G0h1XxKulN+zlGUyuRlgFLhf3t4U3ldYpJ9ZDjgqW3Y7teulwLUtCfTD /VnlwalgQGiFZbmlj07JbUX6Sp54fAN7SxMoxkItZT8pyryAeS9XmoLet1E+NORYJiar iYyZW8dCc0lMzuXcFL3U4QE+FrlZuAGGckHXyur/pD6//ulYUoFzkf8zDD0aROPp40k/ xmTrCxx3e3E9BLeb2nij/r9kqOJAFZfyDKOq1UGUdyyy0zgGTjg9yGYmpyEbnJHJf0ZV W5qYg5oFjenWEcOuTpD8cYePEcPbJSlpmAYb7YzYWjMWXvtbqAw/EQzvEU6N9zSkiTXv C/RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682960158; x=1685552158; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+1Ptlsu3HrvWcynZoOK0ZqIIaOTUmOtwKZY5J1VaLvs=; b=dPUUfiLBxHFG9jk1dOBl4XR7k+ID3cz0YKvRMn/sB+08iQPOfrOrfpG0/q5xnN6XDp ZolvvexSaSwEAibmR8rKU1e3xadukb5vMioCWVYSimKZLasdpRgjy/nP+5e/MGgaZtGf Y5GXDEtn0tlcZLxt2lzOLN5+NmcF/6slUPoPIdrlRpVFe3xSPJtHvXElcsHAy6x0PkCn RyYNvsCNj0bDvtq4mF40oSAjg4E1suSHQ7YgtPo3clwHNrZ4419kxdbvUbg4lsvnNmSJ JKylnZCTtufi1KXjOCBv6JItUHIauxMB2WiHihU75rgKUAZQFPng6zuq1TfSjjpahoCe SiGw== X-Gm-Message-State: AC+VfDwiNZcPgztKX8eo7/FR7Ag9CY8gscKK1LpU99FxJSTOmnazy0GX 6SLGgvwxtOaEXA4uMC7dlCK6au2JFds= X-Google-Smtp-Source: ACHHUZ7r1ycZ9itNH1qfXqxuYbQFcFPivnhoEDNh4tWwK8chvFvWCKKjG1ErV0+LZWXbi1M27uOVRiApR9A= X-Received: from surenb-desktop.mtv.corp.google.com ([2620:15c:211:201:6d24:3efd:facc:7ac4]) (user=surenb job=sendgmr) by 2002:a05:6902:1028:b0:b8c:607:7669 with SMTP id x8-20020a056902102800b00b8c06077669mr8930549ybt.5.1682960158430; Mon, 01 May 2023 09:55:58 -0700 (PDT) Date: Mon, 1 May 2023 09:54:32 -0700 In-Reply-To: <20230501165450.15352-1-surenb@google.com> Mime-Version: 1.0 References: <20230501165450.15352-1-surenb@google.com> X-Mailer: git-send-email 2.40.1.495.gc816e09b53d-goog Message-ID: <20230501165450.15352-23-surenb@google.com> Subject: [PATCH 22/40] mm: create new codetag references during page splitting From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, mhocko@suse.com, vbabka@suse.cz, hannes@cmpxchg.org, roman.gushchin@linux.dev, mgorman@suse.de, dave@stgolabs.net, willy@infradead.org, liam.howlett@oracle.com, corbet@lwn.net, void@manifault.com, peterz@infradead.org, juri.lelli@redhat.com, ldufour@linux.ibm.com, catalin.marinas@arm.com, will@kernel.org, arnd@arndb.de, tglx@linutronix.de, mingo@redhat.com, dave.hansen@linux.intel.com, x86@kernel.org, peterx@redhat.com, david@redhat.com, axboe@kernel.dk, mcgrof@kernel.org, masahiroy@kernel.org, nathan@kernel.org, dennis@kernel.org, tj@kernel.org, muchun.song@linux.dev, rppt@kernel.org, paulmck@kernel.org, pasha.tatashin@soleen.com, yosryahmed@google.com, yuzhao@google.com, dhowells@redhat.com, hughd@google.com, andreyknvl@gmail.com, keescook@chromium.org, ndesaulniers@google.com, gregkh@linuxfoundation.org, ebiggers@google.com, ytcoode@gmail.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, bristot@redhat.com, vschneid@redhat.com, cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, 42.hyeyoo@gmail.com, glider@google.com, elver@google.com, dvyukov@google.com, shakeelb@google.com, songmuchun@bytedance.com, jbaron@akamai.com, rientjes@google.com, minchan@google.com, kaleshsingh@google.com, surenb@google.com, kernel-team@android.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arch@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-modules@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5A22C40022 X-Rspam-User: X-Stat-Signature: pbaatzeeznk6z86s6g1qzyb1qe5tm444 X-HE-Tag: 1682960159-424855 X-HE-Meta: U2FsdGVkX1/5eFFd+0Po8Z/kTbTvFyVrjG+HNC0eKq0ijcKhm3Sba5Fbecho7l/1Xu8CrZgIGnOungelcZjV2ZNk/hcaochwh0BE9pot6igH20QXHBcgOR5Jg3pFW+8oGNmArhbzb5UxRg02pG1SYzYk8/ksrMQMg6+ao+J17sQKCzwJqKWIR2/mhPcBv9X3vN3o9tBXO8oRmZPgyoBx1DdqxQGPFi/sYmO+jmnaqu/nw1Acx1gSNF3z5KZww0NlBkUVlyQKdCaNcpboavCC7hTtizJfFGe7h8EgWOcTuT4DsWqmY+h7rd8WPls94yFxqpFU8YnuLp26SIZgwB8JR+Hh0LdW4PyoANGTZDdqJSXeQVlzO1J+PQP7ANMu/Cfz//Evgi1Z5+0UcXf1Lw58r4WTY3+Jhro6ogT4rs6yMh+ciNei+ZAcgnqtOz3eAvaV4uV5W6uNoHOxI8ihOhY5PhnbjwhUMz9BhiONziMHk1Hk7G9fAWRSjGbVljPJxHFo7a/0WEMSCJd50S903tu681oAuBB3QItSMEdTkU01W62qazXuqaPRXfpNccAEupI/1Vh8iWrSevs6BiTwrMdEErBBg9GF1KoFw3Lt5Hy946Gwj9PbLscPRD7aQSC9Fhmr4qDnfurgOct+t/gQGkTFkuZbfeSBWClYb0Fm+uv83TiaoNPblaPz7pgWq7kh+N4MHMGvvMv/F7z3xVZiMdQbtWkqOczmoDKCJ9r8yJJJUKwpm5RZdNv6vSWfC4QsA8gBH7o35vSjdIyP/PfX34E8S2vL41UGA9B5Et0iJ4AcxkHaWxN9LlV4zFWSVq6iKRN4X+YEqQY3wAllZCTVp82upZ2xPKQ/+C/5bfiNFnYqLQ1x6zjMb3O6Oc80R2VSTGuVjm7KdBuRnpIw+EQWWzcG/l4ysmEhhH6FSO7WdfeJ9p20dGIuz6Izigh2nyRcE5CRIL0IiqL0PigqER7D0eH IL3ilE12 yzxO5XOXIM/mCEbZ3ppdQHKDLRE6JKE//MI9dMglqcjg4moxjBpyHO+GccpeNYDAqZUjLseVtbR+gqD7FBdDfXwicnDmkHLY0M06Akbh7fe2FbreIOytWerEMPBui4n71KSlsABj/JRvji7g2bLD5fsD2VMok05/2ehkv8f5f/v8bxZARJhlk3xugJPqcV7imrMryTlGlWkGIZomFoTCJ7MSNOaFMDPE2y9f77DkiXqpnCB5ypOnxHbzDdZNVLUPB+wdmNOWZIMMhYyn8VEcEkVRgIO+vfQrm3oZWq3X8xyz2GOpaCIUFHVpOydxBw5xp9qMqr3lkqmQ3GGEWrK50UNhOXNtPLLH7c/OQMpDH2rqXhOX5jEZ0ctVFvnKhaQTapDL+ushrNUyMX8o+GMBWXDFrgYyoTTuDQObr7JmdDfgbzC3Spk/zyZLoUubnFZ2cNaXDUi7eBKqeAyHCDg0Ahm9RaushdosxekSP54AONKYMARrli/CXaNtLi7lU92Y7M05SAkyR6u8RNcDtOQ4Z92MpQ2Z/IQ/SQDA9XZGkAKJ2jN2Q3Jv6C3Ih3lOv+BpZQLMqyteb8EcpgeC2anQ9Ikm+lfi8r4RP9veLZCGSfeHVCBqHGbXWYakIlO72/Uu3pwccnIBUD4T7IrZEP0OLuv9TyhC52Z0VTtL2owgw8HSlDS0sc7AOVxtpIQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a high-order page is split into smaller ones, each newly split page should get its codetag. The original codetag is reused for these pages but it's recorded as 0-byte allocation because original codetag already accounts for the original high-order allocated page. Signed-off-by: Suren Baghdasaryan --- include/linux/pgalloc_tag.h | 30 ++++++++++++++++++++++++++++++ mm/huge_memory.c | 2 ++ mm/page_alloc.c | 2 ++ 3 files changed, 34 insertions(+) diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 567327c1c46f..0cbba13869b5 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -52,11 +52,41 @@ static inline void pgalloc_tag_dec(struct page *page, unsigned int order) } } +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) +{ + int i; + struct page_ext *page_ext; + union codetag_ref *ref; + struct alloc_tag *tag; + + if (!mem_alloc_profiling_enabled()) + return; + + page_ext = page_ext_get(page); + if (unlikely(!page_ext)) + return; + + ref = codetag_ref_from_page_ext(page_ext); + if (!ref->ct) + goto out; + + tag = ct_to_alloc_tag(ref->ct); + page_ext = page_ext_next(page_ext); + for (i = 1; i < nr; i++) { + /* New reference with 0 bytes accounted */ + alloc_tag_add(codetag_ref_from_page_ext(page_ext), tag, 0); + page_ext = page_ext_next(page_ext); + } +out: + page_ext_put(page_ext); +} + #else /* CONFIG_MEM_ALLOC_PROFILING */ static inline union codetag_ref *get_page_tag_ref(struct page *page) { return NULL; } static inline void put_page_tag_ref(union codetag_ref *ref) {} #define pgalloc_tag_dec(__page, __size) do {} while (0) +static inline void pgalloc_tag_split(struct page *page, unsigned int nr) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 624671aaa60d..221cce0052a2 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -37,6 +37,7 @@ #include #include #include +#include #include #include @@ -2557,6 +2558,7 @@ static void __split_huge_page(struct page *page, struct list_head *list, /* Caller disabled irqs, so they are still disabled here */ split_page_owner(head, nr); + pgalloc_tag_split(head, nr); /* See comment in __split_huge_page_tail() */ if (PageAnon(head)) { diff --git a/mm/page_alloc.c b/mm/page_alloc.c index edd35500f7f6..8cf5a835af7f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2796,6 +2796,7 @@ void split_page(struct page *page, unsigned int order) for (i = 1; i < (1 << order); i++) set_page_refcounted(page + i); split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); } EXPORT_SYMBOL_GPL(split_page); @@ -5012,6 +5013,7 @@ static void *make_alloc_exact(unsigned long addr, unsigned int order, struct page *last = page + nr; split_page_owner(page, 1 << order); + pgalloc_tag_split(page, 1 << order); split_page_memcg(page, 1 << order); while (page < --last) set_page_refcounted(last);