From patchwork Wed Jun 9 11:38:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32646C48BD1 for ; Wed, 9 Jun 2021 11:39:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D306D60FD8 for ; Wed, 9 Jun 2021 11:39:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D306D60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F6FC6B007B; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 665506B008A; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0480D6B0081; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0096.hostedemail.com [216.40.44.96]) by kanga.kvack.org (Postfix) with ESMTP id A455C6B007B for ; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 23573BA10 for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) X-FDA: 78233990310.15.5F9FB61 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf29.hostedemail.com (Postfix) with ESMTP id D866D548 for ; Wed, 9 Jun 2021 11:39:28 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7F3091FD64; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YZTiZ2krokRZPz74ttwlqY3E9OPfJw2MVLDApZp5kJ0=; b=LDp5gkLsDT1plakKx1QbWTNmHjfTbVB/ZEIRrspp9/GdglbQNNmsfwGjZkEpn/Sxy7U7/U zy0rjY96XUhFuNvSNBUY45+z6MUydML6SOyuq1HAgqSj1C7hvXRl13bX+p27x0FPiHEk72 drEpQmA9Kgi6O5+zbkdunga9lqCMy3A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YZTiZ2krokRZPz74ttwlqY3E9OPfJw2MVLDApZp5kJ0=; b=z65b+i1Nn4+r+AFVNc7LKQ3aYAOq7ZlkVGPjGctN+mrDVuLJ9vgCJ260MIpvYEty25LiyI O/d/ETQxIENCwTCw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 507FE118DD; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YZTiZ2krokRZPz74ttwlqY3E9OPfJw2MVLDApZp5kJ0=; b=LDp5gkLsDT1plakKx1QbWTNmHjfTbVB/ZEIRrspp9/GdglbQNNmsfwGjZkEpn/Sxy7U7/U zy0rjY96XUhFuNvSNBUY45+z6MUydML6SOyuq1HAgqSj1C7hvXRl13bX+p27x0FPiHEk72 drEpQmA9Kgi6O5+zbkdunga9lqCMy3A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YZTiZ2krokRZPz74ttwlqY3E9OPfJw2MVLDApZp5kJ0=; b=z65b+i1Nn4+r+AFVNc7LKQ3aYAOq7ZlkVGPjGctN+mrDVuLJ9vgCJ260MIpvYEty25LiyI O/d/ETQxIENCwTCw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id OPssE3WowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:33 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 10/34] mm, slub: restructure new page checks in ___slab_alloc() Date: Wed, 9 Jun 2021 13:38:39 +0200 Message-Id: <20210609113903.1421-11-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=LDp5gkLs; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=z65b+i1N; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=LDp5gkLs; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=z65b+i1N; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: xrznmbetg3cnkpdsr3s1opegwcg48izg X-Rspamd-Queue-Id: D866D548 X-HE-Tag: 1623238768-683446 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When we allocate slab object from a newly acquired page (from node's partial list or page allocator), we usually also retain the page as a new percpu slab. There are two exceptions - when pfmemalloc status of the page doesn't match our gfp flags, or when the cache has debugging enabled. The current code for these decisions is not easy to follow, so restructure it and add comments. The new structure will also help with the following changes. No functional change. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman --- mm/slub.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index ca7cfc7706cc..0d5c253a39b2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2722,13 +2722,29 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, c->page = page; check_new_page: - if (likely(!kmem_cache_debug(s) && pfmemalloc_match(page, gfpflags))) - goto load_freelist; - /* Only entered in the debug case */ - if (kmem_cache_debug(s) && - !alloc_debug_processing(s, page, freelist, addr)) - goto new_slab; /* Slab failed checks. Next slab needed */ + if (kmem_cache_debug(s)) { + if (!alloc_debug_processing(s, page, freelist, addr)) + /* Slab failed checks. Next slab needed */ + goto new_slab; + else + /* + * For debug case, we don't load freelist so that all + * allocations go through alloc_debug_processing() + */ + goto return_single; + } + + if (unlikely(!pfmemalloc_match(page, gfpflags))) + /* + * For !pfmemalloc_match() case we don't load freelist so that + * we don't make further mismatched allocations easier. + */ + goto return_single; + + goto load_freelist; + +return_single: deactivate_slab(s, page, get_freepointer(s, freelist), c); return freelist;