From patchwork Wed Jun 9 11:38:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0996C48BDF for ; Wed, 9 Jun 2021 11:40:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 56CD060FD8 for ; Wed, 9 Jun 2021 11:40:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 56CD060FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 21E536B007E; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3C526B0082; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0CED6B0083; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0195.hostedemail.com [216.40.44.195]) by kanga.kvack.org (Postfix) with ESMTP id 600706B0089 for ; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0F7F9824999B for ; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) X-FDA: 78233990352.05.D1E6752 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf13.hostedemail.com (Postfix) with ESMTP id EC18BE000253 for ; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B2A47219A0; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tMRaEZFMqbqCquk4QWerCUuNLFI/dmGIIEo12matNCQ=; b=JixlPVZ/dhqgk2DKFXTb9/92CPbymu2U2qdnX76atqNlB3g4f5zRYLrlagVZykKDgTLiih vdtQ2g8HU6gvUexrcHtoj++v92tsyscySr+X6yT768ECqYJ9UoxdH2tvJvzE81A8ZjkZJl DiBFYN5o+iGwciHTh9oG8bqhimuctos= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tMRaEZFMqbqCquk4QWerCUuNLFI/dmGIIEo12matNCQ=; b=zfxQHLpAV+0vuOAM4N2A1QLCJuhr4WM5p+V9/v0tkidBV+a2F3kGHCnPj1z3qv/ypM1tUg uCtQ1DyLEASbacBw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 85851118DD; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tMRaEZFMqbqCquk4QWerCUuNLFI/dmGIIEo12matNCQ=; b=JixlPVZ/dhqgk2DKFXTb9/92CPbymu2U2qdnX76atqNlB3g4f5zRYLrlagVZykKDgTLiih vdtQ2g8HU6gvUexrcHtoj++v92tsyscySr+X6yT768ECqYJ9UoxdH2tvJvzE81A8ZjkZJl DiBFYN5o+iGwciHTh9oG8bqhimuctos= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tMRaEZFMqbqCquk4QWerCUuNLFI/dmGIIEo12matNCQ=; b=zfxQHLpAV+0vuOAM4N2A1QLCJuhr4WM5p+V9/v0tkidBV+a2F3kGHCnPj1z3qv/ypM1tUg uCtQ1DyLEASbacBw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id gI0YIHaowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:34 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 16/34] mm, slub: validate slab from partial list or page allocator before making it cpu slab Date: Wed, 9 Jun 2021 13:38:45 +0200 Message-Id: <20210609113903.1421-17-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="JixlPVZ/"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=zfxQHLpA; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="JixlPVZ/"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=zfxQHLpA; spf=pass (imf13.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: EC18BE000253 X-Stat-Signature: mo4bg8787w3z1rwocw5jcwwuf4xghqpb X-HE-Tag: 1623238771-697610 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When we obtain a new slab page from node partial list or page allocator, we assign it to kmem_cache_cpu, perform some checks, and if they fail, we undo the assignment. In order to allow doing the checks without irq disabled, restructure the code so that the checks are done first, and kmem_cache_cpu.page assignment only after they pass. Signed-off-by: Vlastimil Babka --- mm/slub.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index f4ce372e3dd3..378f1cb040b3 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2729,10 +2729,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, lockdep_assert_irqs_disabled(); freelist = get_partial(s, gfpflags, node, &page); - if (freelist) { - c->page = page; + if (freelist) goto check_new_page; - } local_irq_restore(flags); put_cpu_ptr(s->cpu_slab); @@ -2745,9 +2743,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, } local_irq_save(flags); - if (c->page) - flush_slab(s, c); - /* * No other reference to the page yet so we can * muck around with it freely without cmpxchg @@ -2756,14 +2751,12 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, page->freelist = NULL; stat(s, ALLOC_SLAB); - c->page = page; check_new_page: if (kmem_cache_debug(s)) { if (!alloc_debug_processing(s, page, freelist, addr)) { /* Slab failed checks. Next slab needed */ - c->page = NULL; local_irq_restore(flags); goto new_slab; } else { @@ -2782,10 +2775,18 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, */ goto return_single; + if (unlikely(c->page)) + flush_slab(s, c); + c->page = page; + goto load_freelist; return_single: + if (unlikely(c->page)) + flush_slab(s, c); + c->page = page; + deactivate_slab(s, page, get_freepointer(s, freelist), c); local_irq_restore(flags); return freelist;