From patchwork Fri Mar 8 03:13:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 13586441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52775C5475B for ; Fri, 8 Mar 2024 03:13:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB1856B0325; Thu, 7 Mar 2024 22:13:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D3AA56B0327; Thu, 7 Mar 2024 22:13:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDBA56B0328; Thu, 7 Mar 2024 22:13:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AE9306B0325 for ; Thu, 7 Mar 2024 22:13:43 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5180AC126B for ; Fri, 8 Mar 2024 03:13:43 +0000 (UTC) X-FDA: 81872401926.05.96D44A9 Received: from out-186.mta1.migadu.com (out-186.mta1.migadu.com [95.215.58.186]) by imf28.hostedemail.com (Postfix) with ESMTP id 96B07C0008 for ; Fri, 8 Mar 2024 03:13:41 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="DTSbIW/y"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf28.hostedemail.com: domain of kent.overstreet@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709867622; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=ft9pudsb5Zfd645rRWJdMLUAQctehtBPKqFnOZAzYdU=; b=gjZc5hWgf/YfMli2YyYBqGm3uqq2tIh5LsqYXQNIVAV6pnW634AeuvrQ9q4dpRRoop3DZp Nzts8cQAyku4GAmoEg1iurcoBEaOVojQewvhn3ZVkFxnr6acIC81NEutRuSMR4zJzSN5I6 /RhshuUD73OYqz/4H2K6xOcb8Nb5VLk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="DTSbIW/y"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf28.hostedemail.com: domain of kent.overstreet@linux.dev designates 95.215.58.186 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709867622; a=rsa-sha256; cv=none; b=S0/LtwuLNo9+Svq4HQ0VcDho+YdjPvzyblc2OBHqLrvk4iMZRmUK1oLVIt3uiy6heR6dhE j2TF47K1veIMdjwizeo8z5dX6ib9MbWmypyyd+Dy8UX0XhBkFm9WXmIHt2BNkVi6uH5HZ1 sCkutmLJQBnEV8ZYI+U66cK/acEyekA= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1709867619; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=ft9pudsb5Zfd645rRWJdMLUAQctehtBPKqFnOZAzYdU=; b=DTSbIW/yXq6KfRFyTiKEPF81qj/gCmMJfrZ4G0zXg89RalMX0M4kgEwZ7HamEu/QvUXGzZ HMcvjzwNUAHK0MgfIN0GPcIhFcVQ9DuBemeDf0yUpjwsuXkmBwkuH4iZXCO+v0iuculqR8 djn0/YWjps6U/zNsAp1fYkRJyLvrSJI= From: Kent Overstreet To: Cc: Kent Overstreet , linux-mm@kvack.org, Vlastimil Babka , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Matthew Wilcox Subject: [PATCH] mm/slub: Make __ksize() faster Date: Thu, 7 Mar 2024 22:13:22 -0500 Message-ID: <20240308031325.218676-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 96B07C0008 X-Stat-Signature: 6mr3mk6weqe7pcx1urujbo1dwzhexe5x X-HE-Tag: 1709867621-514716 X-HE-Meta: U2FsdGVkX1+YzwGoTRqyYacVXG956ryfoAZzCC0+AkFa13BTbnqp6lL7fldQZtZx+sa7cO/2Jk6srzRqufr69qAYwiScwxcZa6ndB4QDNcMyC46xsZIvcFtRyRz4E65wwgQmcefl5d/67Ccg6heahnf05Tc2a8392Uq5hTlqRH6I060mwTx7/9FAvGJ7MVYmDufxGwY7pS03xbd0M6tI+VTpsZd8y4TdCn4yiuT0BuNRS5pUwTISqyZ3q98rJHxZnySiCcSTkmxUmm0PDVCLCpvhwbCFuCya2uy5UU1V1gGiV6ZWTB3VBpkreyCELn4WGY7l7EB1DXhqTEeJA1QvZ6Bh+1hrWg5hZySDQpZ5nu7WQs/VOO3phdgWI8ENcxJn6BENy2M1Aqk8UgYBrXjNPbCxR2DhVyuaQlZ+9bYOQZllcV3lIBTOXjQRMb7GFjO8Rzs3QvcdY0XfOnlDfDW7Ku1BeCo5JTtoMAPTIj5yefZbPY2k1jrcRqFyT+jZw+Mz83zszGp7ZB9hCeIknCAvrEkOZRmOn3H1mi3JDgZXgYfTQCyzbtUfkHTXfZIXE8ufnXZYDydLhB7o1xvIYtassdt/acD93W+AnmcSaZA5Pw6Zih6MMOSgkF0raiDDUmh2UWuWY8WhmZCrUop4DrZIH4YM8Monezymobat2q6iwp+4aL//882pMc1NP5O/56k1kssCPdnsuJTpW9fRViqG2kJp6HiX16FTP6UJj4vWXkhkq9XnhT20HW6elHRa7UtucUi3eRlI/1xKuwPBN9cwOjlb/zxD8f5OwyGTTa+5W3bRA9uLJGCsYtsoOQer80iCVuK8vzAgXbCnyHzsvAhi/B91aUcim0wqk8JX6fi02myzLa6MN5kJfR5rUdcoVlYMPF5HQKlP+uBw6LMk1lsaUQrr9m0KyfR9A3ybiLXkH6PWpiTnN1TwbfUqJqwTVd3Ql7j5B+kcPag= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: this is part of a larger in-progress patch series to add tracking of amount of memory stranded via RCU. I want to base this on willy's series that rearranges folio batch freeing - that's got __folio_put() cleanups I want. Once he's got that up in a git tree I'll do that and finish the rest. -- >8 -- with slab gone, we now have a free u32 in struct slab. This steals it to make __ksize() faster; it's now a single dependent load, instead of two. This is going to be important for tracking the amount of memory stranded by RCU, which we want to be able to do if we're going to be freeing all pagecache folios (and perhaps all folios) via RCU. Signed-off-by: Kent Overstreet Cc: linux-mm@kvack.org Cc: Vlastimil Babka Cc: Andrew Morton Cc: Christoph Lameter Cc: Pekka Enberg Cc: David Rientjes Cc: Roman Gushchin Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: Matthew Wilcox (Oracle) --- mm/slab.h | 2 +- mm/slab_common.c | 9 ++++----- mm/slub.c | 1 + 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/slab.h b/mm/slab.h index 54deeb0428c6..64f06431cc97 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -84,7 +84,7 @@ struct slab { }; struct rcu_head rcu_head; }; - unsigned int __unused; + unsigned int object_size; atomic_t __page_refcount; #ifdef CONFIG_MEMCG diff --git a/mm/slab_common.c b/mm/slab_common.c index 6ec0f6543f34..f209b8cf4965 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -963,13 +963,12 @@ size_t __ksize(const void *object) if (WARN_ON(object != folio_address(folio))) return 0; return folio_size(folio); - } - + } else { #ifdef CONFIG_SLUB_DEBUG - skip_orig_size_check(folio_slab(folio)->slab_cache, object); + skip_orig_size_check(folio_slab(folio)->slab_cache, object); #endif - - return slab_ksize(folio_slab(folio)->slab_cache); + return folio_slab(folio)->object_size; + } } /** diff --git a/mm/slub.c b/mm/slub.c index 2ef88bbf56a3..37fe5774c110 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2366,6 +2366,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) } slab->objects = oo_objects(oo); + slab->object_size = slab_ksize(slab->slab_cache); slab->inuse = 0; slab->frozen = 0;