From patchwork Sat Sep 4 10:49:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12475511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 320F1C433EF for ; Sat, 4 Sep 2021 10:50:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id DB7B760FD7 for ; Sat, 4 Sep 2021 10:50:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org DB7B760FD7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id E15566B007E; Sat, 4 Sep 2021 06:50:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A5E066B0081; Sat, 4 Sep 2021 06:50:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AC55940009; Sat, 4 Sep 2021 06:50:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0226.hostedemail.com [216.40.44.226]) by kanga.kvack.org (Postfix) with ESMTP id 94F816B007B for ; Sat, 4 Sep 2021 06:50:13 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 635522FE14 for ; Sat, 4 Sep 2021 10:50:13 +0000 (UTC) X-FDA: 78549571506.28.8E6C26D Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf04.hostedemail.com (Postfix) with ESMTP id 087DC50000A1 for ; Sat, 4 Sep 2021 10:50:12 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 1661322693; Sat, 4 Sep 2021 10:50:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1630752612; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7mUlsuwJHAvWrWwqdtLB62kmN3qgv5TJrxQ2cxDjcmE=; b=BBSUyAXX5kguKuXcpd4Zj1oL/He0ejBI+SXoza9DHncYVVNpjlTy9zGdJdcZDBpta6SpJk ISDdoW5N3ioBayQOzVgjcFK9vco1QLOAq8ldv952sAuyKIFR4u/5PzYlig6Z6uHCLhy5UK +w5UH01Xmb9r4j7EAXB+GQldAAZBVLs= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1630752612; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7mUlsuwJHAvWrWwqdtLB62kmN3qgv5TJrxQ2cxDjcmE=; b=YrY9Y4GaOV9p7Pgd7d7+0I31YZxo9L2gLFiZ+N4X/ZC+notpR3SH91NoAFRxu8aRVmG/iN YbR+aa+M8dweV9Cw== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id E244713A2C; Sat, 4 Sep 2021 10:50:11 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id qD6hNmNPM2HoUQAAMHmgww (envelope-from ); Sat, 04 Sep 2021 10:50:11 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Andrew Morton , linux-kernel@vger.kernel.org, Mike Galbraith , Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Vlastimil Babka Subject: [PATCH v6 15/33] mm, slub: validate slab from partial list or page allocator before making it cpu slab Date: Sat, 4 Sep 2021 12:49:45 +0200 Message-Id: <20210904105003.11688-16-vbabka@suse.cz> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20210904105003.11688-1-vbabka@suse.cz> References: <20210904105003.11688-1-vbabka@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=2080; h=from:subject; bh=6RMKkRPIFqUauCH8SJwma9svu+hJUqHG71X1n1a6wH8=; b=owEBbQGS/pANAwAIAeAhynPxiakQAcsmYgBhM08tI9N51RCa6f2o+EKv7HZaeUBnieZWGaRsW7io r6kzmKKJATMEAAEIAB0WIQSNS5MBqTXjGL5IXszgIcpz8YmpEAUCYTNPLQAKCRDgIcpz8YmpEFWHB/ oCpGr5YOouRz5tfSzXgWhHJK8AbYmNHL3IL9+OF0nrlOp91ELjXLFYcblzjoNzKoxjM7f8BWiVsA2P gdKi8lXtY+w0Bhl6l86tgO4reK1Gy9o6P5+Uwo4e0LcBV0ujx+h2pOkTNeD6PkEFcTf+FtiRs+Q6T2 GvwOHRh9hvoNiyHKbuoQJOXXjpjB1j/EiEwfjbp7VfCTx4d+XZGRZAfvlrCNcZoNx2ikR7baf1uiQ1 u9YQGAHLotozScAnM33PIrBzZr89vCYEPrIx4eac+MfaYnGNSdLt2sS3+v61oOLfWuQOqkb3PP/oIA PzpItUyBr1HK0LhAVIyH6F6T+2BsNw X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=BBSUyAXX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=YrY9Y4Ga; spf=pass (imf04.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: x66zub9ro5rwtuw914dt98iwzh8iwgr7 X-Rspamd-Queue-Id: 087DC50000A1 X-Rspamd-Server: rspam04 X-HE-Tag: 1630752612-670186 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When we obtain a new slab page from node partial list or page allocator, we assign it to kmem_cache_cpu, perform some checks, and if they fail, we undo the assignment. In order to allow doing the checks without irq disabled, restructure the code so that the checks are done first, and kmem_cache_cpu.page assignment only after they pass. Signed-off-by: Vlastimil Babka --- mm/slub.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 7798ba1c614f..a5e974defcb7 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2802,10 +2802,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, lockdep_assert_irqs_disabled(); freelist = get_partial(s, gfpflags, node, &page); - if (freelist) { - c->page = page; + if (freelist) goto check_new_page; - } local_irq_restore(flags); put_cpu_ptr(s->cpu_slab); @@ -2818,9 +2816,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, } local_irq_save(flags); - if (c->page) - flush_slab(s, c); - /* * No other reference to the page yet so we can * muck around with it freely without cmpxchg @@ -2829,14 +2824,12 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, page->freelist = NULL; stat(s, ALLOC_SLAB); - c->page = page; check_new_page: if (kmem_cache_debug(s)) { if (!alloc_debug_processing(s, page, freelist, addr)) { /* Slab failed checks. Next slab needed */ - c->page = NULL; local_irq_restore(flags); goto new_slab; } else { @@ -2855,10 +2848,18 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, */ goto return_single; + if (unlikely(c->page)) + flush_slab(s, c); + c->page = page; + goto load_freelist; return_single: + if (unlikely(c->page)) + flush_slab(s, c); + c->page = page; + deactivate_slab(s, page, get_freepointer(s, freelist), c); local_irq_restore(flags); return freelist;