From patchwork Mon Mar 17 14:33:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 14019361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1335C35FF9 for ; Mon, 17 Mar 2025 14:33:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 461D0280004; Mon, 17 Mar 2025 10:33:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2EC6A280008; Mon, 17 Mar 2025 10:33:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D0B8280006; Mon, 17 Mar 2025 10:33:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BFBB9280004 for ; Mon, 17 Mar 2025 10:33:49 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8D9CB1C688A for ; Mon, 17 Mar 2025 14:33:50 +0000 (UTC) X-FDA: 83231287020.11.B900B64 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf26.hostedemail.com (Postfix) with ESMTP id 7036F140009 for ; Mon, 17 Mar 2025 14:33:48 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742222028; a=rsa-sha256; cv=none; b=6GtGlYAZ5z3IqSGEywf0Ay1xMIJlqkiw6xIJKsYHLFOFHOdFM1xDlCGXtlPmVlG8n6rgOI dmcMZi4AWtEYZ4lPUbX68jz9E4/JNIDnin/PhjeOBT3X0tOy3olS70CEWxAxHLo6lJtnzU J95UBaXYfnszvEOhJT7VNjkSXalAG48= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742222028; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AiTIBAssu+u2dt5mFFaQ9oy7Lmw9BsLyAhD/T45P1Dg=; b=KehxPcscdtL28/faXe8/lfOxNEYONPMrLNN2GcWTEReMfkHq+pVjYo0cUhUWEETMpPVVtb Q5F0gMR7LRCS3J9TXYMMg6QmMLhEq1MR4asMYzfOPToVk5TB/NkZYpae/Qa7aHBYgjYJ2n wYPjZj9hdquY+jqcTHoRx7Dz/wMn0Mw= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 10A1821AF7; Mon, 17 Mar 2025 14:33:34 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id F040A13A5D; Mon, 17 Mar 2025 14:33:33 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id eK1jOr0y2GcycQAAD6G6ig (envelope-from ); Mon, 17 Mar 2025 14:33:33 +0000 From: Vlastimil Babka Date: Mon, 17 Mar 2025 15:33:06 +0100 Subject: [PATCH RFC v3 5/8] slab: determine barn status racily outside of lock MIME-Version: 1.0 Message-Id: <20250317-slub-percpu-caches-v3-5-9d9884d8b643@suse.cz> References: <20250317-slub-percpu-caches-v3-0-9d9884d8b643@suse.cz> In-Reply-To: <20250317-slub-percpu-caches-v3-0-9d9884d8b643@suse.cz> To: Suren Baghdasaryan , "Liam R. Howlett" , Christoph Lameter , David Rientjes Cc: Roman Gushchin , Harry Yoo , Uladzislau Rezki , linux-mm@kvack.org, linux-kernel@vger.kernel.org, rcu@vger.kernel.org, maple-tree@lists.infradead.org, vbabka@suse.cz X-Mailer: b4 0.14.2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Rspamd-Server: rspam07 X-Rspam-User: X-Stat-Signature: xachc7n55asq3fgfcybi7yao7bwuaqhs X-Rspamd-Queue-Id: 7036F140009 X-HE-Tag: 1742222028-646102 X-HE-Meta: U2FsdGVkX1+Q/Qn6MaTJuKsln1b8/HaJDkZEf7rOeDZp4Wg8w3cdBE89U3FvBtySEIUzFJdyPHxeUPWeLQyZxUpL+a1lTVpg5CNV5rJSPX+HDSIQgeri942DGfuhdCszXEuE0o86fJ6cM4skQxveV801jtxIzlISt0Tuyzpl6mzAQLAx577KYdUsOSeXHafusQknHhmkLdDjFwVYdR+xT5f0tbgJHg3WaYP8paDkelIraJt5ituiojrjHogznl8m3jaW+yWE3/9mMEWCINdn3nY3vAIzEe2btmL8eiGE2j23DcG3Vo/IXtPAkPngRTgJEL3fGtDFH8kvObylJyOHn3FeatBE/BTzW/14ExCR3NisIYBaZv+JOIe9jX4/sQaD/WD14Ka7jHzeguCxWlPrEYVtwm+3OBz5IgtdCFF1mH980v2laHpC+1a5uMPXqp2oJQV4BIwwdvJFe0jj+Kplt0MydZuBMo5fLpZFigh62vYGUKBpGEIcYHmJQiRx1OX9oeeNAIkNEmbQYZ6LNWAsdIZ7I++KrDnpXJRO0rksPZRdI+0lDtnF2+5NQwTnY/ZiYU0DUnJ7k20JUFDPEKEDhttQd+fZiRXqnHJs4PycIWht+T7fWdIV9S/A9uxIuYC+V8GNkjdCdw3uqs/qsCPLjLDo3vJXdILBWVfmlLABc8QXfTF8+RTwiOJKTO+rgQhBiItenMR4LlxpHHUgimxZotRTU0sNTu0Zu5nMUdMyEOqx3vuJylUwLBwYerq7J1wsZJwA2VLh93Kt+Qt4VZOAYRBW0fKTanP3ocS8LVcy7K1mIAu0M8TKicCAZB1zx2Fd175jWq8EqTBpMwWDGGmoMPlMhU8oksNdscAHfxATUohFI9lXfv90fRxhM07g93JxitPpupF46+xozJxz06dQ6qBa/sKGKxzZt53TmNjhSroCfE/fW/37DXJfn7DzilwIVulGIoBqG6oSdZg6MlG utKhouxx 34xJ7suiVAUuld4dBZ1y9HZ5dwBPClCgpALevWFbSmot9rfcSFgRQX+161V8XgqP0Y8TnlcYDmgz2n5rAat6Fzr0Csc4dq2Qejdm+08ncqVpuK4Kz+BFH9uirrI6MUhtgDgE9CZ9cv+XjGndFYxs9l/lbb5PxFRAsWvBu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The possibility of many barn operations is determined by the current number of full or empty sheaves. Taking the barn->lock just to find out that e.g. there are no empty sheaves results in unnecessary overhead and lock contention. Thus perform these checks outside of the lock with a data_race() annotated variable read and fail quickly without taking the lock. Checks for sheaf availability that racily succeed have to be obviously repeated under the lock for correctness, but we can skip repeating checks if there are too many sheaves on the given list as the limits don't need to be strict. Signed-off-by: Vlastimil Babka Reviewed-by: Suren Baghdasaryan Reviewed-by: Harry Yoo --- mm/slub.c | 57 ++++++++++++++++++++++++++++++++++----------------------- 1 file changed, 34 insertions(+), 23 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index ab3532d5f41045d8268b12ad774541dcd066c4c4..2c7b2a85c628d01fa4811dd6cd3e4fd198d80381 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2698,9 +2698,12 @@ static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn) struct slab_sheaf *empty = NULL; unsigned long flags; + if (!data_race(barn->nr_empty)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); - if (barn->nr_empty) { + if (likely(barn->nr_empty)) { empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf, barn_list); list_del(&empty->barn_list); @@ -2716,38 +2719,36 @@ static int barn_put_empty_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf, bool ignore_limit) { unsigned long flags; - int ret = 0; + + /* we don't repeat the check under barn->lock as it's not critical */ + if (!ignore_limit && data_race(barn->nr_empty) >= MAX_EMPTY_SHEAVES) + return -E2BIG; spin_lock_irqsave(&barn->lock, flags); - if (!ignore_limit && barn->nr_empty >= MAX_EMPTY_SHEAVES) { - ret = -E2BIG; - } else { - list_add(&sheaf->barn_list, &barn->sheaves_empty); - barn->nr_empty++; - } + list_add(&sheaf->barn_list, &barn->sheaves_empty); + barn->nr_empty++; spin_unlock_irqrestore(&barn->lock, flags); - return ret; + return 0; } static int barn_put_full_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf, bool ignore_limit) { unsigned long flags; - int ret = 0; + + /* we don't repeat the check under barn->lock as it's not critical */ + if (!ignore_limit && data_race(barn->nr_full) >= MAX_FULL_SHEAVES) + return -E2BIG; spin_lock_irqsave(&barn->lock, flags); - if (!ignore_limit && barn->nr_full >= MAX_FULL_SHEAVES) { - ret = -E2BIG; - } else { - list_add(&sheaf->barn_list, &barn->sheaves_full); - barn->nr_full++; - } + list_add(&sheaf->barn_list, &barn->sheaves_full); + barn->nr_full++; spin_unlock_irqrestore(&barn->lock, flags); - return ret; + return 0; } static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn) @@ -2755,6 +2756,9 @@ static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn) struct slab_sheaf *sheaf = NULL; unsigned long flags; + if (!data_race(barn->nr_full) && !data_race(barn->nr_empty)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); if (barn->nr_full) { @@ -2785,9 +2789,12 @@ barn_replace_empty_sheaf(struct node_barn *barn, struct slab_sheaf *empty) struct slab_sheaf *full = NULL; unsigned long flags; + if (!data_race(barn->nr_full)) + return NULL; + spin_lock_irqsave(&barn->lock, flags); - if (barn->nr_full) { + if (likely(barn->nr_full)) { full = list_first_entry(&barn->sheaves_full, struct slab_sheaf, barn_list); list_del(&full->barn_list); @@ -2810,19 +2817,23 @@ barn_replace_full_sheaf(struct node_barn *barn, struct slab_sheaf *full) struct slab_sheaf *empty; unsigned long flags; + /* we don't repeat this check under barn->lock as it's not critical */ + if (data_race(barn->nr_full) >= MAX_FULL_SHEAVES) + return ERR_PTR(-E2BIG); + if (!data_race(barn->nr_empty)) + return ERR_PTR(-ENOMEM); + spin_lock_irqsave(&barn->lock, flags); - if (barn->nr_full >= MAX_FULL_SHEAVES) { - empty = ERR_PTR(-E2BIG); - } else if (!barn->nr_empty) { - empty = ERR_PTR(-ENOMEM); - } else { + if (likely(barn->nr_empty)) { empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf, barn_list); list_del(&empty->barn_list); list_add(&full->barn_list, &barn->sheaves_full); barn->nr_empty--; barn->nr_full++; + } else { + empty = ERR_PTR(-ENOMEM); } spin_unlock_irqrestore(&barn->lock, flags);