From patchwork Sat Sep 4 10:49:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12475497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 303C3C433F5 for ; Sat, 4 Sep 2021 10:50:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D4EB760F21 for ; Sat, 4 Sep 2021 10:50:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D4EB760F21 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 5BE9D94000B; Sat, 4 Sep 2021 06:50:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B6F466B0078; Sat, 4 Sep 2021 06:50:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 862BC94000E; Sat, 4 Sep 2021 06:50:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0218.hostedemail.com [216.40.44.218]) by kanga.kvack.org (Postfix) with ESMTP id 5FCA294000A for ; Sat, 4 Sep 2021 06:50:12 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 201B418559318 for ; Sat, 4 Sep 2021 10:50:12 +0000 (UTC) X-FDA: 78549571464.22.1918909 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf12.hostedemail.com (Postfix) with ESMTP id CF38110000A3 for ; Sat, 4 Sep 2021 10:50:11 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id DF17520044; Sat, 4 Sep 2021 10:50:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1630752610; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VoLeY7+Afh9ZsA0cNFKiMXjj0drfqsWveV98Vysevbg=; b=2fF62YLk7YC+bAb0nPZRwt0YxwwxQZdGVQStYVQQVAIP6f0Aa88P4CgGiwByBgmfbXNTJ2 toUbgOXThELdFOIMpYiL75hM48hWWay5siEMT+nVwdNT9vbuglJnRfojbYGE14R3TTntpE YBsyO8LwtW0y5xENg1C6aU9nv0a8I7c= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1630752610; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VoLeY7+Afh9ZsA0cNFKiMXjj0drfqsWveV98Vysevbg=; b=JZqlwOzIno7o+2BlexDjOVfc5W3Mn+2L3Ec006p6RC5os8DAP1JE8TqUqwA1sG8VkJBMVO oLeorT1lY2MplaAQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id B545713A2C; Sat, 4 Sep 2021 10:50:10 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id sM6vK2JPM2HoUQAAMHmgww (envelope-from ); Sat, 04 Sep 2021 10:50:10 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Andrew Morton , linux-kernel@vger.kernel.org, Mike Galbraith , Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Vlastimil Babka Subject: [PATCH v6 09/33] mm, slub: restructure new page checks in ___slab_alloc() Date: Sat, 4 Sep 2021 12:49:39 +0200 Message-Id: <20210904105003.11688-10-vbabka@suse.cz> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20210904105003.11688-1-vbabka@suse.cz> References: <20210904105003.11688-1-vbabka@suse.cz> MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1913; h=from:subject; bh=cTI28CzoWvLSfFayxzYm7bnXwtvYONmwMSO/dMD+k9Y=; b=owEBbQGS/pANAwAIAeAhynPxiakQAcsmYgBhM08dw3VbU8f/S9qM3NoZnUyP9HUuUrw1ci/9KFa6 4HUwRI2JATMEAAEIAB0WIQSNS5MBqTXjGL5IXszgIcpz8YmpEAUCYTNPHQAKCRDgIcpz8YmpED6LCA CUXfBIWK3C5femaiBytbdi9JRjQIZpgV//CwmiHknD6r8H+wMLPSQ+IS5s4/Il5Jqocj5OdwKO4l6F ZAWjq+1/aJfL4M3775HeJ5ugfSkyzKHrNWieQxWkvnnUrK0E8ZOk/0EWRsO+fnFcAEopfZac/bDxLA I2EoIGessG6x08tqZPVOAy6KzuDd5EoSTYWvLSO3aAe+fqMZ5BCISsTfU++X3HxUZBs4b+jwB3abOF n0/7Eg4fFlRdjMPyPFP5o/zE8sh6sCT0VPrq6rl7TvqES8ArVOtQEByIlgM6HuV3MePK90KvScx7Fg oenaCKc+otjPS6dGJJaK0chHdW9Rxs X-Developer-Key: i=vbabka@suse.cz; a=openpgp; fpr=A940D434992C2E8E99103D50224FA7E7CC82A664 Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=2fF62YLk; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=JZqlwOzI; spf=pass (imf12.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: 6qn6ib6t1bf3j3igifbra3k6g6hnssru X-Rspamd-Queue-Id: CF38110000A3 X-Rspamd-Server: rspam04 X-HE-Tag: 1630752611-938823 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When we allocate slab object from a newly acquired page (from node's partial list or page allocator), we usually also retain the page as a new percpu slab. There are two exceptions - when pfmemalloc status of the page doesn't match our gfp flags, or when the cache has debugging enabled. The current code for these decisions is not easy to follow, so restructure it and add comments. The new structure will also help with the following changes. No functional change. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman --- mm/slub.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index e9d582eee7d7..9607ce37e661 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2782,13 +2782,29 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, c->page = page; check_new_page: - if (likely(!kmem_cache_debug(s) && pfmemalloc_match(page, gfpflags))) - goto load_freelist; - /* Only entered in the debug case */ - if (kmem_cache_debug(s) && - !alloc_debug_processing(s, page, freelist, addr)) - goto new_slab; /* Slab failed checks. Next slab needed */ + if (kmem_cache_debug(s)) { + if (!alloc_debug_processing(s, page, freelist, addr)) + /* Slab failed checks. Next slab needed */ + goto new_slab; + else + /* + * For debug case, we don't load freelist so that all + * allocations go through alloc_debug_processing() + */ + goto return_single; + } + + if (unlikely(!pfmemalloc_match(page, gfpflags))) + /* + * For !pfmemalloc_match() case we don't load freelist so that + * we don't make further mismatched allocations easier. + */ + goto return_single; + + goto load_freelist; + +return_single: deactivate_slab(s, page, get_freepointer(s, freelist), c); return freelist;