From patchwork Thu Oct 24 16:23:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Suren Baghdasaryan X-Patchwork-Id: 13849365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AB68CE8E92 for ; Thu, 24 Oct 2024 16:23:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B48676B00A2; Thu, 24 Oct 2024 12:23:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF9EA6B00A3; Thu, 24 Oct 2024 12:23:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9995E6B00A4; Thu, 24 Oct 2024 12:23:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 74D726B00A2 for ; Thu, 24 Oct 2024 12:23:24 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C74C016139F for ; Thu, 24 Oct 2024 16:23:02 +0000 (UTC) X-FDA: 82709014668.20.ADA733D Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf01.hostedemail.com (Postfix) with ESMTP id 7725B40007 for ; Thu, 24 Oct 2024 16:23:07 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gDaWUl1d; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3eXQaZwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3eXQaZwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729786950; a=rsa-sha256; cv=none; b=L7OevDk/Y7am1Sf7ghA6Yn9qG1H+ktPng1m/AMsAHJ8IdTNoqa+CdMXmyItC3oiPgqU/6K 0KcCC2o15aWBk8sm8aEGpeL4TTDzgo6TjjZEOMKL0PPt+ZseSp0d01l4k8lWpHAY6LEQTw EMxlrrNpatpnQoNbVr4ccJ2cSCctYWU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gDaWUl1d; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of 3eXQaZwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3eXQaZwYKCAw463qzns00sxq.o0yxuz69-yyw7mow.03s@flex--surenb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729786950; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=gTU/Az6oQC87qV0R2JNeB98NQAAeDAB9smYAoMmnxCc=; b=sEcatz5UAKCTDGSPNCmoTg1aNrXrA4VT495BF9A79bIQ4dM2z33tz7PwnDaoHSoUIGEW0P Rvntwnz4qWJyFeQUNCqN4KlPvqVjepV6QYzKr7iFOb78Q9QobISLR4uQBi5QAofbApISBz VmhevhWW9y50aIN8KPY4VpdxMjeNtAY= Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6e321d26b38so18387997b3.2 for ; Thu, 24 Oct 2024 09:23:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729787001; x=1730391801; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=gTU/Az6oQC87qV0R2JNeB98NQAAeDAB9smYAoMmnxCc=; b=gDaWUl1d+FY3Xz3oYlPiIaR3iPmrEhXgNDt8bc0MxGcsqdpOSzBIxQayEQmaKxBivK ti7b6IMlTQVqG8f130Of7KNO8+a8iOatHLZad3Way7l8hFm5y3w/faWpHx3S5pB5QmwX LPX79ZvSH7VLlrYMC8E3ZVAPvK4iJaN4bep9VNymxsztp8MHGGfGsja7V61MsZggSewN 7GzRua4cYIrOJjUmu9WOQw3JGrX7DbFIrsAHQZvilvzDH0pWu1SDLaNmIblYeDRdOpwh SmaIPfeNXeafdbmiZSpuFu+5gnvSZD0rv/2d5q1UgUcNGNOMTC6wY1XBLsXdUuqb228n n97Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729787001; x=1730391801; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gTU/Az6oQC87qV0R2JNeB98NQAAeDAB9smYAoMmnxCc=; b=SZRNQGjSjWtpqtwpa+/inlp1QLfXfLpW95SPAruHJ6kajY4fXo8LRhDRQNevxkoqdj gz1M2Sp5tL6dXK2G7J+0XSElYwvqvYn/YTMCNLdzyHWhpv9qpmvqNQ+O/cfXt7/MgxQt ZYd7TVsVWbCN0lRlXLdZoEJDFjbfwf5tAg4doEsH0/viORWdZEIvppuryg6csl/kJYsx hmxuYHrdo9EHwqm4KuXytzYFZenm4Wuv5KtH0a21yrKgIVThJued+B9PtbHbEeiURFQq yxiB5PrvZeEPvktXMU4Bw/2kYT2yK4GFYQFCkUlW1Z4KAxo8bfTUNb5TBnqRkHSTUwia 8AsQ== X-Forwarded-Encrypted: i=1; AJvYcCW3HIVQldQSYWtn4ilGnN5iapn8KUAC5Vtfxq6SmE4efJUmwSmZqELWQnDdEgkginze1Fje9r1DeA==@kvack.org X-Gm-Message-State: AOJu0Yz9NO5h3nzXUQseHYcC6dPhG1zU/r8BbV3aal8hVkCD2UFKBZlW dDKrx1CqyP1CSm8mdI2hwRf4G8JO/d0hic8gml63r0sO+vrCOs+wG1VAbtGpzLzm6KN+5lwLb4C 5KA== X-Google-Smtp-Source: AGHT+IGzDL8ww+0/uK6UkUgn7P+6nNxCybhx2NYsrl5o+4THb/v3434ldbCSMuGjsNZGJegfhEedS/IaArE= X-Received: from surenb-desktop.mtv.corp.google.com ([2a00:79e0:2e3f:8:47e4:54e:bc8e:cb18]) (user=surenb job=sendgmr) by 2002:a05:690c:3802:b0:6de:19f:34d7 with SMTP id 00721157ae682-6e858143817mr695977b3.2.1729787001130; Thu, 24 Oct 2024 09:23:21 -0700 (PDT) Date: Thu, 24 Oct 2024 09:23:18 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.0.105.g07ac214952-goog Message-ID: <20241024162318.1640781-1-surenb@google.com> Subject: [PATCH 1/1] mm/codetag: uninline and move pgalloc_tag_copy and pgalloc_tag_split From: Suren Baghdasaryan To: akpm@linux-foundation.org Cc: kent.overstreet@linux.dev, yuzhao@google.com, souravpanda@google.com, pasha.tatashin@soleen.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, surenb@google.com X-Rspam-User: X-Rspamd-Queue-Id: 7725B40007 X-Rspamd-Server: rspam01 X-Stat-Signature: fjia6t9dw3ks93wp3pyx4kunxzzocr65 X-HE-Tag: 1729786987-978641 X-HE-Meta: U2FsdGVkX18EbPMvMJGHRKQ4CHAfpqiJIqcGYnADoA0x22wQmO+UQoHJEfrrLSJwmMIM1Ivan2AbbYmRG3kQSeNQdv+n0ndCXAqI0CPNtuaKEL6CneCOCIvq7TUUcOXnK2s1NtSymeSZM/mgieIaXldAS2E8kEh6+CIv3gDp4wuSluHCSKWF4LrwWtgR8JLO1zXEiMYnylL98qbSs1ybymYZxAoBl2hdyTxy4pO5gImZ/5m1jrmAja+s0Lb1q8evB5AMGVhtOqPcnV7EKf0J70HEOpyCGu4oAejSXDJAVHG/7l5l4M2sP3+toCJ0lYjB36VXnr/HbA6yJpnk/0y/XZwmyY9zHPVB0/r8erUJAIUCrql2rV2qBE4N6FAt8ljaimc1GU2g5yyN2LdqzLgNXfy/XOGSWfc3M2hyoeyA7auMCudo+JizqUoYVpqZiJROTGaV/tY8y6m9iFr3Qhci85O24adw8N4SCP2NFQ4RimQ4Q+YAAdbrh8C9NWzxxl03AFQRS87742MlcFE4IF/ru/5a9i6puiFpm1gNe4YWHiytilqukttxbgf8jmkehHdYp8wO+qob5AYv6XW1G9lGEBBcts6FBLkuQffnh8OrVCaXL/hNpKwHdtg4hYyWp0SrptyKEykEK7qJl+UqhSjuV56Fj2R3+MGTDycxA64dWIrXvDIpXeCd6SC3KHluuWr5HhEsK1pd9Udkclte2emAUwuIF3Ypv6+Pvk2h/saQTWlhVb8aAUV1qp8DX7zMIf6EBl8n79BZ6UZvVpN/JTHjyGCqY9DeNBYN1OW4zTQr6X3jy2P9oh/KWoDxTN50XvVat8voZW/Du0Jrk0P/Rk+K6P4wBl+EetUJ25g93cm3phcoPSEXg+Rd5zWd4AeitudcdbekxQ6msr5rMR9oLcC3Az/j3xy5aMh7ThC6wRpQXggO4UmYYjYWETY3t5YF5XoHcIDEy+NKjV9+x/vY8+q 5OkJVtdm ozp8TFa6DPniRKbGMq+2aO7qkBp7r4tVgoL99IUwgG8uT7rDlwlCFw+IN3nkgSprqiI6gWHzCLLnbl25s5vodMTBWnIYveijz8utKcqxwjkZcqvcDUigs5pjJ9xE5FzDGvgy7GLRpXF6FlyACY1mnoqSmgHsCDSJmmmDMgPgZ7ODG7XxeNJPtpzYcoT0eLKzskH93k1kO8BgtIrauH4U7RE0HpBW8tOODtXV54j7rRRQ/Jw+4CvwZkREfz63YaaGU+pNtLhFQdKM1+FnSpCk2zAZRo6lDBaz+vYSpcaJctdFJrdxJlHYhgKQ38ABMMMCrFruOZbZCo5CV+bjhJwjl3vp42cHo7UNHVwDAG8bxl+imJMmi+tF/zWTvMzmVGxT++Ica6etFcP2Q1s3kYsARjeroDuYsRdFfgwfxDjOJMelMT9WMOCvzhNqcYBBYrHQzouznYAko+fgtKtODuMyS2HRGfG18b4XO7UT4tMxHWLKw61DKj6JinICWm+xmFe0fNZjaOpecq++lLNB57Cp39cfymgDf2TDyS3N597sZ4Qx93SGsrg6cOChd6L2TTrVrMrWTjg/j+RXAZK8rOsG5XMFYiTVpDY93X3BxzXsBqu9kMGIaN7+1yfpyOnK5tv9nxXOn/YIhZfuaUfU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000040, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: pgalloc_tag_copy() and pgalloc_tag_split() are sizable and outside of any performance-critical paths, so it should be fine to uninline them. Also move their declarations into pgalloc_tag.h which seems like a more appropriate place for them. No functional changes other than uninlining. Signed-off-by: Suren Baghdasaryan Suggested-by: Andrew Morton Acked-by: Yu Zhao --- Applies over mm-unstable include/linux/mm.h | 58 ------------------------------------- include/linux/pgalloc_tag.h | 5 ++++ lib/alloc_tag.c | 48 ++++++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 58 deletions(-) base-commit: 9c111059234a949a4d3442a413ade19cc65ab927 diff --git a/include/linux/mm.h b/include/linux/mm.h index 4ef8cf1043f1..5184624c0f21 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4165,62 +4165,4 @@ static inline int do_mseal(unsigned long start, size_t len_in, unsigned long fla } #endif -#ifdef CONFIG_MEM_ALLOC_PROFILING -static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) -{ - int i; - struct alloc_tag *tag; - unsigned int nr_pages = 1 << new_order; - - if (!mem_alloc_profiling_enabled()) - return; - - tag = pgalloc_tag_get(&folio->page); - if (!tag) - return; - - for (i = nr_pages; i < (1 << old_order); i += nr_pages) { - union pgtag_ref_handle handle; - union codetag_ref ref; - - if (get_page_tag_ref(folio_page(folio, i), &ref, &handle)) { - /* Set new reference to point to the original tag */ - alloc_tag_ref_set(&ref, tag); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); - } - } -} - -static inline void pgalloc_tag_copy(struct folio *new, struct folio *old) -{ - union pgtag_ref_handle handle; - union codetag_ref ref; - struct alloc_tag *tag; - - tag = pgalloc_tag_get(&old->page); - if (!tag) - return; - - if (!get_page_tag_ref(&new->page, &ref, &handle)) - return; - - /* Clear the old ref to the original allocation tag. */ - clear_page_tag_ref(&old->page); - /* Decrement the counters of the tag on get_new_folio. */ - alloc_tag_sub(&ref, folio_size(new)); - __alloc_tag_ref_set(&ref, tag); - update_page_tag_ref(handle, &ref); - put_page_tag_ref(handle); -} -#else /* !CONFIG_MEM_ALLOC_PROFILING */ -static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) -{ -} - -static inline void pgalloc_tag_copy(struct folio *new, struct folio *old) -{ -} -#endif /* CONFIG_MEM_ALLOC_PROFILING */ - #endif /* _LINUX_MM_H */ diff --git a/include/linux/pgalloc_tag.h b/include/linux/pgalloc_tag.h index 1fe63b52e5e5..0e43ab653ab6 100644 --- a/include/linux/pgalloc_tag.h +++ b/include/linux/pgalloc_tag.h @@ -230,6 +230,9 @@ static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) this_cpu_sub(tag->counters->bytes, PAGE_SIZE * nr); } +void pgalloc_tag_split(struct folio *folio, int old_order, int new_order); +void pgalloc_tag_copy(struct folio *new, struct folio *old); + void __init alloc_tag_sec_init(void); #else /* CONFIG_MEM_ALLOC_PROFILING */ @@ -241,6 +244,8 @@ static inline void pgalloc_tag_sub(struct page *page, unsigned int nr) {} static inline struct alloc_tag *pgalloc_tag_get(struct page *page) { return NULL; } static inline void pgalloc_tag_sub_pages(struct alloc_tag *tag, unsigned int nr) {} static inline void alloc_tag_sec_init(void) {} +static inline void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) {} +static inline void pgalloc_tag_copy(struct folio *new, struct folio *old) {} #endif /* CONFIG_MEM_ALLOC_PROFILING */ diff --git a/lib/alloc_tag.c b/lib/alloc_tag.c index a6f6f014461e..c1ddac2d29f0 100644 --- a/lib/alloc_tag.c +++ b/lib/alloc_tag.c @@ -163,6 +163,54 @@ size_t alloc_tag_top_users(struct codetag_bytes *tags, size_t count, bool can_sl return nr; } +void pgalloc_tag_split(struct folio *folio, int old_order, int new_order) +{ + int i; + struct alloc_tag *tag; + unsigned int nr_pages = 1 << new_order; + + if (!mem_alloc_profiling_enabled()) + return; + + tag = pgalloc_tag_get(&folio->page); + if (!tag) + return; + + for (i = nr_pages; i < (1 << old_order); i += nr_pages) { + union pgtag_ref_handle handle; + union codetag_ref ref; + + if (get_page_tag_ref(folio_page(folio, i), &ref, &handle)) { + /* Set new reference to point to the original tag */ + alloc_tag_ref_set(&ref, tag); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); + } + } +} + +void pgalloc_tag_copy(struct folio *new, struct folio *old) +{ + union pgtag_ref_handle handle; + union codetag_ref ref; + struct alloc_tag *tag; + + tag = pgalloc_tag_get(&old->page); + if (!tag) + return; + + if (!get_page_tag_ref(&new->page, &ref, &handle)) + return; + + /* Clear the old ref to the original allocation tag. */ + clear_page_tag_ref(&old->page); + /* Decrement the counters of the tag on get_new_folio. */ + alloc_tag_sub(&ref, folio_size(new)); + __alloc_tag_ref_set(&ref, tag); + update_page_tag_ref(handle, &ref); + put_page_tag_ref(handle); +} + static void shutdown_mem_profiling(bool remove_file) { if (mem_alloc_profiling_enabled())