From patchwork Fri May 31 09:33:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 13681424 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1745BC25B75 for ; Fri, 31 May 2024 09:33:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 943D66B009A; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F11F6B009F; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 743936B009B; Fri, 31 May 2024 05:33:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5644D6B0095 for ; Fri, 31 May 2024 05:33:44 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0168BA1CE1 for ; Fri, 31 May 2024 09:33:43 +0000 (UTC) X-FDA: 82178178768.04.072C46A Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf30.hostedemail.com (Postfix) with ESMTP id CBBC280010 for ; Fri, 31 May 2024 09:33:41 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717148022; a=rsa-sha256; cv=none; b=5sHt+3lKlN8Bf+IcLW2oYc+uwJKSE7loId04scdWJMFQFyFQ+IYR8EwnV74r4OWdH/aqam /GTPzlknN6XwVIcH4fGpmLAS6leyjRma8kcJUHHU51J+f359FYwuZBuc94ZKzz0lJxEHnb 0MYsMnphlQ+tGUz4PPsCaBpWb2hh0oA= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717148022; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MZP7fHUs/rbLHpmHtvQS8St+zXrEp6ICcSeYQ5zEHn4=; b=xB5RW1uNNC4RFk3y8eEvKNgQl4BhfzrKj0qx4+JX3M+zlQH4IS9yqJQcpgO2AKg6yLewEZ hYhU8SS1AwP/5T7tt6Y9mxuwefTSARC9PCsvdFZOhFeRgJ2KB+bIswJbXhS2XcDUgvgN2z qoGmEw6ppqH1qXLgmWpS9Y5ue5Gn73o= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 49B0E21B2B; Fri, 31 May 2024 09:33:40 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 2907813A91; Fri, 31 May 2024 09:33:40 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id 2I+7CXSZWWZKHQAAD6G6ig (envelope-from ); Fri, 31 May 2024 09:33:40 +0000 From: Vlastimil Babka Date: Fri, 31 May 2024 11:33:34 +0200 Subject: [PATCH RFC 3/4] mm, slab: add static key for should_failslab() MIME-Version: 1.0 Message-Id: <20240531-fault-injection-statickeys-v1-3-a513fd0a9614@suse.cz> References: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> In-Reply-To: <20240531-fault-injection-statickeys-v1-0-a513fd0a9614@suse.cz> To: Akinobu Mita , Christoph Lameter , David Rientjes , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , "Naveen N. Rao" , Anil S Keshavamurthy , "David S. Miller" , Masami Hiramatsu , Steven Rostedt , Mark Rutland Cc: Jiri Olsa , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, linux-trace-kernel@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.13.0 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CBBC280010 X-Stat-Signature: 8f6yddmgrmm76o59o9zc9n1ddfw7k64f X-HE-Tag: 1717148021-883203 X-HE-Meta: U2FsdGVkX1/X1wjibFM6CHwqxGbYpl/jpcUHCRZ+jTNXAhrcF5C/+UI5C660JRj8QLPIIdHb3f/ZTfnwvLK5tCsWnJ3cdvJRBIglmeJfwNCajXDcqNklquJR48DjV+bjo40j0PvWJwgQNw0S4lRsP0oqGPoZqp2tIcIShYx1b8kKHCP04MjiU+cyJBdWVS5iR78IzCAVDlW5o+Fzvi0HrhodjxdvYQ5AZWJ+GIvoLJZum2gzZvHr2iy9UvoULnEZFfMfpi4H14mg65D8k/ScdMeJKfTP7bbtmyeTdQLCKdCAo/5pLfgfN5XaTaA0tfP1ra2NYVyVx46xIv+HLiyN6UH4tnoXzangLITlKMzRkfQ1nVtTryCAnTDV4mTCIXk6gomoIcOmbHzKrm7so9g/4PoDMYfoHG68pFeuPBNVZhbr2FQMAfFdJBch4EQzs//j9spskJyDsPSAHAzEOS8/WNTErfuD/EZKrVFqYR72DJRIxjExrtkga3XJtHfTvL6sIwtYPmEuapk9Zg3uiQgTZa2X/zfRJ2yU8fr+sGLhZ3XrqIKWBY8yih055DEdiXZp2w3xTkbL9Xap6dzPphl/ww2NILZGqc6CuFIs0nRD8XGSyG3dxl2G+LAUiPhXIi/fCvT61evcZTiduMupTTBygPzvKDiFUkXmciPMrg4Jvxcx4a8FLjFXSC2PvojHD989ELVF0gtQ5L+WMH1cGHTB45JKQ+krRKroY13zc9Yp/ZiHsgJmg529usWb5wbIi8ImyLKTLd2RAsuhOM6nACJq2dHf9Z95r2HNNDfHFvbKT8ZK3UPB3ymZ9QuEAxb+ua/a+zTrtsPru4rWRTanhhhFXlIyxpvO0R5+p6zUMNx+Xavx4z7NNZg53vpcaTADSnkUQ14m94GPwpqOc9wDm6lLJ/vv5GF6z+ot/fzc/o5XBDCClayut/ft3tauq+wK9Guxf9q0+fo/jH9kjBRGkKT q309pR8d G0NxOGDPqQx3JqqTsDlHOlJXhTMZThtMWDQ/u3uck3c8MNwTvPQtZ0WnOmbI7KNc2lGuFvhlhStq+AvnVZ+1AHNbAljtV9qfWkE+witj4kpHs+6QIV3A1J0eWjrWcw+KC6CskqnWtLE3vmXn586gqtwhmVr2Pxvt94WbBTKf7NTiVql5mXNBDPJYTlbEIMNCp2y1WpbhTxEMWfP4S4FfHqf5eyCy9XZtguhJ/x9uKm7g32oo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Since commit 4f6923fbb352 ("mm: make should_failslab always available for fault injection") should_failslab() is unconditionally a noinline function. This adds visible overhead to the slab allocation hotpath, even if the function is empty. With CONFIG_FAILSLAB=y there's additional overhead when the functionality is not enabled by a boot parameter or debugfs. The overhead can be eliminated with a static key around the callsite. Fault injection and error injection frameworks can now be told that the this function has a static key associated, and are able to enable and disable it accordingly. Signed-off-by: Vlastimil Babka Acked-by: Alexei Starovoitov Reviewed-by: Roman Gushchin --- mm/failslab.c | 2 +- mm/slab.h | 3 +++ mm/slub.c | 10 +++++++--- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/mm/failslab.c b/mm/failslab.c index ffc420c0e767..878fd08e5dac 100644 --- a/mm/failslab.c +++ b/mm/failslab.c @@ -9,7 +9,7 @@ static struct { bool ignore_gfp_reclaim; bool cache_filter; } failslab = { - .attr = FAULT_ATTR_INITIALIZER, + .attr = FAULT_ATTR_INITIALIZER_KEY(&should_failslab_active.key), .ignore_gfp_reclaim = true, .cache_filter = false, }; diff --git a/mm/slab.h b/mm/slab.h index 5f8f47c5bee0..792e19cb37b8 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -11,6 +11,7 @@ #include #include #include +#include /* * Internal slab definitions @@ -160,6 +161,8 @@ static_assert(IS_ALIGNED(offsetof(struct slab, freelist), sizeof(freelist_aba_t) */ #define slab_page(s) folio_page(slab_folio(s), 0) +DECLARE_STATIC_KEY_FALSE(should_failslab_active); + /* * If network-based swap is enabled, sl*b must keep track of whether pages * were allocated from pfmemalloc reserves. diff --git a/mm/slub.c b/mm/slub.c index 0809760cf789..3bb579760a37 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3874,13 +3874,15 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, 0, sizeof(void *)); } +DEFINE_STATIC_KEY_FALSE(should_failslab_active); + noinline int should_failslab(struct kmem_cache *s, gfp_t gfpflags) { if (__should_failslab(s, gfpflags)) return -ENOMEM; return 0; } -ALLOW_ERROR_INJECTION(should_failslab, ERRNO); +ALLOW_ERROR_INJECTION_KEY(should_failslab, ERRNO, &should_failslab_active); static __fastpath_inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags) @@ -3889,8 +3891,10 @@ struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags) might_alloc(flags); - if (unlikely(should_failslab(s, flags))) - return NULL; + if (static_branch_unlikely(&should_failslab_active)) { + if (should_failslab(s, flags)) + return NULL; + } return s; }