From patchwork Wed Jun 9 11:38:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64CEDC48BD1 for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 138BA613C7 for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 138BA613C7 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 94D9D6B0071; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 884106B0036; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B00D6B0075; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0153.hostedemail.com [216.40.44.153]) by kanga.kvack.org (Postfix) with ESMTP id 382CB6B0036 for ; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id CB47D180AD81A for ; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) X-FDA: 78233990226.14.EDC155A Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf01.hostedemail.com (Postfix) with ESMTP id 1131B5001703 for ; Wed, 9 Jun 2021 11:39:28 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B4A19219BC; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238771; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vEDtjsnB/VDzYc0ii5k8c4+YLz6GPrWXLWLa4QOy8Fg=; b=IN8TwQl/nBF3IjB1x1wI2UnNVr4f2PMGC+TK64rkhGBc2qSF8AFsivay09xFcheo9OVNnb EyLzyuilsT1tTjlp2yyxsvWQb58GMdNR+OwaJ0TRA1eK8O/e7IFrccI/kDY37vtaVHZwGL s8ecWHWDT7kuc2g2E1LONThysPQHrZc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238771; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vEDtjsnB/VDzYc0ii5k8c4+YLz6GPrWXLWLa4QOy8Fg=; b=d08Kapwu2aJQczRX1Wiq9dop5yCfAX59yWf5r9X5g8KMXpk96Xqof3b8tqDspWR+X5J7f/ 8lm2LZHonHT3E8Cw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 7022111A98; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238771; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vEDtjsnB/VDzYc0ii5k8c4+YLz6GPrWXLWLa4QOy8Fg=; b=IN8TwQl/nBF3IjB1x1wI2UnNVr4f2PMGC+TK64rkhGBc2qSF8AFsivay09xFcheo9OVNnb EyLzyuilsT1tTjlp2yyxsvWQb58GMdNR+OwaJ0TRA1eK8O/e7IFrccI/kDY37vtaVHZwGL s8ecWHWDT7kuc2g2E1LONThysPQHrZc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238771; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vEDtjsnB/VDzYc0ii5k8c4+YLz6GPrWXLWLa4QOy8Fg=; b=d08Kapwu2aJQczRX1Wiq9dop5yCfAX59yWf5r9X5g8KMXpk96Xqof3b8tqDspWR+X5J7f/ 8lm2LZHonHT3E8Cw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id WLXrGnOowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:31 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 01/34] mm, slub: don't call flush_all() from list_locations() Date: Wed, 9 Jun 2021 13:38:30 +0200 Message-Id: <20210609113903.1421-2-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="IN8TwQl/"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=d08Kapwu; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="IN8TwQl/"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=d08Kapwu; spf=pass (imf01.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: rzjge6hmtpt1hqreyrek7ypk8i9giihm X-Rspamd-Queue-Id: 1131B5001703 X-Rspamd-Server: rspam06 X-HE-Tag: 1623238768-474323 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: list_locations() can only be called on caches with SLAB_STORE_USER flag and as with all slub debugging flags, such caches avoid cpu or percpu partial slabs altogether, so there's nothing to flush. Signed-off-by: Vlastimil Babka --- mm/slub.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 3f96e099817a..f928607230b2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4839,8 +4839,6 @@ static int list_locations(struct kmem_cache *s, char *buf, GFP_KERNEL)) { return sysfs_emit(buf, "Out of memory\n"); } - /* Push back cpu slabs */ - flush_all(s); for_each_kmem_cache_node(s, node, n) { unsigned long flags; From patchwork Wed Jun 9 11:38:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED798C48BCD for ; Wed, 9 Jun 2021 11:39:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9CBCA613D3 for ; Wed, 9 Jun 2021 11:39:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9CBCA613D3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4B0596B0070; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F5EA6B007D; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A80DF6B0074; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id 566096B0072 for ; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id EB51A181AEF21 for ; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) X-FDA: 78233990226.17.C89EF02 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf04.hostedemail.com (Postfix) with ESMTP id 778033C3 for ; Wed, 9 Jun 2021 11:39:30 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E30C81FD4E; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238771; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0yIopAqiYKYuUjHruBmtFqmPxVRpqk1W58WY5m3tQOo=; b=S79B+ZPYD1ZEeUaxt9hKWMzfTD9TOCTuQo++QSf3MvKDtglnvsOS8/YGbi3Xq0JJ6K6lmW dtNx6p12EKQ3My9QOMM53cX1ikrojSZ9U7mnFv7sHMjP04xp6R9yIHmO+Fp8h6njhfUxQ/ rXeGkSO2barA5BNDxVQ6PPyWVxcH9d4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238771; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0yIopAqiYKYuUjHruBmtFqmPxVRpqk1W58WY5m3tQOo=; b=tvWi3K3sT5luoYzW/nK8wj0xXpVpZ8R87g1ey45dq0fCH2nFemnOGde2D5HP1jxG4N3U+L BHQF/jdnl506dGCg== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id B6328118DD; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238771; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0yIopAqiYKYuUjHruBmtFqmPxVRpqk1W58WY5m3tQOo=; b=S79B+ZPYD1ZEeUaxt9hKWMzfTD9TOCTuQo++QSf3MvKDtglnvsOS8/YGbi3Xq0JJ6K6lmW dtNx6p12EKQ3My9QOMM53cX1ikrojSZ9U7mnFv7sHMjP04xp6R9yIHmO+Fp8h6njhfUxQ/ rXeGkSO2barA5BNDxVQ6PPyWVxcH9d4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238771; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0yIopAqiYKYuUjHruBmtFqmPxVRpqk1W58WY5m3tQOo=; b=tvWi3K3sT5luoYzW/nK8wj0xXpVpZ8R87g1ey45dq0fCH2nFemnOGde2D5HP1jxG4N3U+L BHQF/jdnl506dGCg== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id sF8ALHOowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:31 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 02/34] mm, slub: allocate private object map for sysfs listings Date: Wed, 9 Jun 2021 13:38:31 +0200 Message-Id: <20210609113903.1421-3-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=S79B+ZPY; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=tvWi3K3s; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=S79B+ZPY; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=tvWi3K3s; dmarc=none; spf=pass (imf04.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: bs3tgr35qka6mftd1sak6y4obddxwxt6 X-Rspamd-Queue-Id: 778033C3 X-HE-Tag: 1623238770-51391 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Slub has a static spinlock protected bitmap for marking which objects are on freelist when it wants to list them, for situations where dynamically allocating such map can lead to recursion or locking issues, and on-stack bitmap would be too large. The handlers of sysfs files alloc_calls and free_calls also currently use this shared bitmap, but their syscall context makes it straightforward to allocate a private map before entering locked sections, so switch these processing paths to use a private bitmap. Signed-off-by: Vlastimil Babka Acked-by: Christoph Lameter Acked-by: Mel Gorman --- mm/slub.c | 42 ++++++++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 14 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index f928607230b2..92c3ab3a95ba 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -448,6 +448,18 @@ static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page, static unsigned long object_map[BITS_TO_LONGS(MAX_OBJS_PER_PAGE)]; static DEFINE_SPINLOCK(object_map_lock); +static void __fill_map(unsigned long *obj_map, struct kmem_cache *s, + struct page *page) +{ + void *addr = page_address(page); + void *p; + + bitmap_zero(obj_map, page->objects); + + for (p = page->freelist; p; p = get_freepointer(s, p)) + set_bit(__obj_to_index(s, addr, p), obj_map); +} + /* * Determine a map of object in use on a page. * @@ -457,17 +469,11 @@ static DEFINE_SPINLOCK(object_map_lock); static unsigned long *get_map(struct kmem_cache *s, struct page *page) __acquires(&object_map_lock) { - void *p; - void *addr = page_address(page); - VM_BUG_ON(!irqs_disabled()); spin_lock(&object_map_lock); - bitmap_zero(object_map, page->objects); - - for (p = page->freelist; p; p = get_freepointer(s, p)) - set_bit(__obj_to_index(s, addr, p), object_map); + __fill_map(object_map, s, page); return object_map; } @@ -4813,17 +4819,17 @@ static int add_location(struct loc_track *t, struct kmem_cache *s, } static void process_slab(struct loc_track *t, struct kmem_cache *s, - struct page *page, enum track_item alloc) + struct page *page, enum track_item alloc, + unsigned long *obj_map) { void *addr = page_address(page); void *p; - unsigned long *map; - map = get_map(s, page); + __fill_map(obj_map, s, page); + for_each_object(p, s, addr, page->objects) - if (!test_bit(__obj_to_index(s, addr, p), map)) + if (!test_bit(__obj_to_index(s, addr, p), obj_map)) add_location(t, s, get_track(s, p, alloc)); - put_map(map); } static int list_locations(struct kmem_cache *s, char *buf, @@ -4834,9 +4840,15 @@ static int list_locations(struct kmem_cache *s, char *buf, struct loc_track t = { 0, 0, NULL }; int node; struct kmem_cache_node *n; + unsigned long *obj_map; + + obj_map = bitmap_alloc(oo_objects(s->oo), GFP_KERNEL); + if (!obj_map) + return sysfs_emit(buf, "Out of memory\n"); if (!alloc_loc_track(&t, PAGE_SIZE / sizeof(struct location), GFP_KERNEL)) { + bitmap_free(obj_map); return sysfs_emit(buf, "Out of memory\n"); } @@ -4849,12 +4861,14 @@ static int list_locations(struct kmem_cache *s, char *buf, spin_lock_irqsave(&n->list_lock, flags); list_for_each_entry(page, &n->partial, slab_list) - process_slab(&t, s, page, alloc); + process_slab(&t, s, page, alloc, obj_map); list_for_each_entry(page, &n->full, slab_list) - process_slab(&t, s, page, alloc); + process_slab(&t, s, page, alloc, obj_map); spin_unlock_irqrestore(&n->list_lock, flags); } + bitmap_free(obj_map); + for (i = 0; i < t.count; i++) { struct location *l = &t.loc[i]; From patchwork Wed Jun 9 11:38:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A9EFC48BD1 for ; Wed, 9 Jun 2021 11:39:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 00AE5613CC for ; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 00AE5613CC Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id DBCAF6B006E; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B44586B0073; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 836686B0078; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0195.hostedemail.com [216.40.44.195]) by kanga.kvack.org (Postfix) with ESMTP id 485266B0070 for ; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id DF4D782499A8 for ; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) X-FDA: 78233990226.23.162E7A0 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf06.hostedemail.com (Postfix) with ESMTP id 7E487C00CBFC for ; Wed, 9 Jun 2021 11:39:29 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 1FA57219D8; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ep08mPuIvp4EJCRoqvQdo5MzcdTX3TMZH7zztWTIia0=; b=R7fnjYimc679BHUpiF6s6n+TndfMlmv3in5YbEsoWvVbtrIM3mjnwQ/JN6gj05Fd5zjfp1 b+agQmoPzJei/HQeYdPB/oxCe6bPVSDgBxvpb3Q4JP5IUa7uG3tL2EIFicdbX+qw+uwrzP 8hDpPUNN5Ifpc7fnGC5vMWLyc/JIxtk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ep08mPuIvp4EJCRoqvQdo5MzcdTX3TMZH7zztWTIia0=; b=oWylKST498T94BiWwVY8PKNnX8opFCzB2faYnQDQdhR/HtEaAKADRTehjawfbaToeq/Yzp hG5pC1E5+QpdPQBQ== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id E509A11A98; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ep08mPuIvp4EJCRoqvQdo5MzcdTX3TMZH7zztWTIia0=; b=R7fnjYimc679BHUpiF6s6n+TndfMlmv3in5YbEsoWvVbtrIM3mjnwQ/JN6gj05Fd5zjfp1 b+agQmoPzJei/HQeYdPB/oxCe6bPVSDgBxvpb3Q4JP5IUa7uG3tL2EIFicdbX+qw+uwrzP 8hDpPUNN5Ifpc7fnGC5vMWLyc/JIxtk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ep08mPuIvp4EJCRoqvQdo5MzcdTX3TMZH7zztWTIia0=; b=oWylKST498T94BiWwVY8PKNnX8opFCzB2faYnQDQdhR/HtEaAKADRTehjawfbaToeq/Yzp hG5pC1E5+QpdPQBQ== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id gPl2N3OowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:31 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 03/34] mm, slub: allocate private object map for validate_slab_cache() Date: Wed, 9 Jun 2021 13:38:32 +0200 Message-Id: <20210609113903.1421-4-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7E487C00CBFC X-Stat-Signature: 1ax3c94u8i4jun9qwru3h9n116htzioj Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=R7fnjYim; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=oWylKST4; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=R7fnjYim; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=oWylKST4; dmarc=none; spf=pass (imf06.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-HE-Tag: 1623238769-639730 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: validate_slab_cache() is called either to handle a sysfs write, or from a self-test context. In both situations it's straightforward to preallocate a private object bitmap instead of grabbing the shared static one meant for critical sections, so let's do that. Signed-off-by: Vlastimil Babka Acked-by: Christoph Lameter Acked-by: Mel Gorman --- mm/slub.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 92c3ab3a95ba..5a940bd3cebb 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4622,11 +4622,11 @@ static int count_total(struct page *page) #endif #ifdef CONFIG_SLUB_DEBUG -static void validate_slab(struct kmem_cache *s, struct page *page) +static void validate_slab(struct kmem_cache *s, struct page *page, + unsigned long *obj_map) { void *p; void *addr = page_address(page); - unsigned long *map; slab_lock(page); @@ -4634,21 +4634,20 @@ static void validate_slab(struct kmem_cache *s, struct page *page) goto unlock; /* Now we know that a valid freelist exists */ - map = get_map(s, page); + __fill_map(obj_map, s, page); for_each_object(p, s, addr, page->objects) { - u8 val = test_bit(__obj_to_index(s, addr, p), map) ? + u8 val = test_bit(__obj_to_index(s, addr, p), obj_map) ? SLUB_RED_INACTIVE : SLUB_RED_ACTIVE; if (!check_object(s, page, p, val)) break; } - put_map(map); unlock: slab_unlock(page); } static int validate_slab_node(struct kmem_cache *s, - struct kmem_cache_node *n) + struct kmem_cache_node *n, unsigned long *obj_map) { unsigned long count = 0; struct page *page; @@ -4657,7 +4656,7 @@ static int validate_slab_node(struct kmem_cache *s, spin_lock_irqsave(&n->list_lock, flags); list_for_each_entry(page, &n->partial, slab_list) { - validate_slab(s, page); + validate_slab(s, page, obj_map); count++; } if (count != n->nr_partial) @@ -4668,7 +4667,7 @@ static int validate_slab_node(struct kmem_cache *s, goto out; list_for_each_entry(page, &n->full, slab_list) { - validate_slab(s, page); + validate_slab(s, page, obj_map); count++; } if (count != atomic_long_read(&n->nr_slabs)) @@ -4685,10 +4684,17 @@ static long validate_slab_cache(struct kmem_cache *s) int node; unsigned long count = 0; struct kmem_cache_node *n; + unsigned long *obj_map; + + obj_map = bitmap_alloc(oo_objects(s->oo), GFP_KERNEL); + if (!obj_map) + return -ENOMEM; flush_all(s); for_each_kmem_cache_node(s, node, n) - count += validate_slab_node(s, n); + count += validate_slab_node(s, n, obj_map); + + bitmap_free(obj_map); return count; } From patchwork Wed Jun 9 11:38:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71062C48BCD for ; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1F156613CB for ; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1F156613CB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BC9076B0036; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 996216B0070; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 773FD6B0074; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0086.hostedemail.com [216.40.44.86]) by kanga.kvack.org (Postfix) with ESMTP id 3BEE56B006E for ; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CBC05B9E8 for ; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) X-FDA: 78233990226.22.E63E3E5 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf29.hostedemail.com (Postfix) with ESMTP id 8048F54D for ; Wed, 9 Jun 2021 11:39:27 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 4CD57219DC; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zoOHiL7Kvc+iNuZbvKQCtrRD3qZokJ4ArXlEsxyEuKw=; b=UJ0symXRAnt3IWMogw5ez7B8HoZD9xhnBulyzyFv4vHSMLWvwXCmlmOeEYQ8RSFLUIoh3Q yhq5azcwdMvLlfV1T6WsE9P1fc22ViVWkrx0oWQvDRtJw5mVzm4HbTBGkZqgOKY2T3ZS1s aSxcv8e0eSuEt8bhe5Uts2TkKx4od+k= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zoOHiL7Kvc+iNuZbvKQCtrRD3qZokJ4ArXlEsxyEuKw=; b=NdAvi3bIF8OZl40iBlJF/XsvpwTRCUi1nP6B0ZSa0AjDiEltwOlRwE+ie1ErbtF8dL8fuG dfIrpcGEbKUPk8DQ== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 1EBEF118DD; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zoOHiL7Kvc+iNuZbvKQCtrRD3qZokJ4ArXlEsxyEuKw=; b=UJ0symXRAnt3IWMogw5ez7B8HoZD9xhnBulyzyFv4vHSMLWvwXCmlmOeEYQ8RSFLUIoh3Q yhq5azcwdMvLlfV1T6WsE9P1fc22ViVWkrx0oWQvDRtJw5mVzm4HbTBGkZqgOKY2T3ZS1s aSxcv8e0eSuEt8bhe5Uts2TkKx4od+k= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zoOHiL7Kvc+iNuZbvKQCtrRD3qZokJ4ArXlEsxyEuKw=; b=NdAvi3bIF8OZl40iBlJF/XsvpwTRCUi1nP6B0ZSa0AjDiEltwOlRwE+ie1ErbtF8dL8fuG dfIrpcGEbKUPk8DQ== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id wJwKB3SowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:32 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 04/34] mm, slub: don't disable irq for debug_check_no_locks_freed() Date: Wed, 9 Jun 2021 13:38:33 +0200 Message-Id: <20210609113903.1421-5-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UJ0symXR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NdAvi3bI; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=UJ0symXR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=NdAvi3bI; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: d8tw1jixm3a3b8ygep157dky44drccyt X-Rspamd-Queue-Id: 8048F54D X-HE-Tag: 1623238767-46021 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In slab_free_hook() we disable irqs around the debug_check_no_locks_freed() call, which is unnecessary, as irqs are already being disabled inside the call. This seems to be leftover from the past where there were more calls inside the irq disabled sections. Remove the irq disable/enable operations. Mel noted: > Looks like it was needed for kmemcheck which went away back in 4.15 Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman --- mm/slub.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 5a940bd3cebb..2953f6e43cae 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1545,20 +1545,8 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, { kmemleak_free_recursive(x, s->flags); - /* - * Trouble is that we may no longer disable interrupts in the fast path - * So in order to make the debug calls that expect irqs to be - * disabled we need to disable interrupts temporarily. - */ -#ifdef CONFIG_LOCKDEP - { - unsigned long flags; + debug_check_no_locks_freed(x, s->object_size); - local_irq_save(flags); - debug_check_no_locks_freed(x, s->object_size); - local_irq_restore(flags); - } -#endif if (!(s->flags & SLAB_DEBUG_OBJECTS)) debug_check_no_obj_freed(x, s->object_size); From patchwork Wed Jun 9 11:38:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEC3BC48BD1 for ; Wed, 9 Jun 2021 11:39:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9BA9B613B6 for ; Wed, 9 Jun 2021 11:39:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9BA9B613B6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 25AA06B0072; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3A906B0070; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E3C56B007B; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id 576566B0073 for ; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id F032D81F3 for ; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) X-FDA: 78233990226.24.6B43BA8 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf14.hostedemail.com (Postfix) with ESMTP id 5C6CCC00CBF2 for ; Wed, 9 Jun 2021 11:39:29 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7C1571FD58; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5JmSnpyHqca5PLmH1BHqymRQUyP4QdjzJkGjzwDILP8=; b=RJCMgcjXUe7ROekNc9knNZ3EovAV5bzgtjaRIZ52W6GxVogbwC36wO3UdMhVOohqu9CDr/ mT19g+aZtuBYkD29fLKOUzc9lca6x4UBRSpC3+Xu/MZreTJLiJdxpK4JQo2Di9H85ngtam FuCcTosNSujvBrbU0FWZwuXQyvjED6E= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5JmSnpyHqca5PLmH1BHqymRQUyP4QdjzJkGjzwDILP8=; b=bcbC7k8pol07TGi2Zq6kgTNF6P2X5P72t6Y5S5tDOI2PqAHl7iE1wo2HwFsauNM7dOdkpu cI0/YVvVKf6m+eDA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 4E48311A98; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5JmSnpyHqca5PLmH1BHqymRQUyP4QdjzJkGjzwDILP8=; b=RJCMgcjXUe7ROekNc9knNZ3EovAV5bzgtjaRIZ52W6GxVogbwC36wO3UdMhVOohqu9CDr/ mT19g+aZtuBYkD29fLKOUzc9lca6x4UBRSpC3+Xu/MZreTJLiJdxpK4JQo2Di9H85ngtam FuCcTosNSujvBrbU0FWZwuXQyvjED6E= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5JmSnpyHqca5PLmH1BHqymRQUyP4QdjzJkGjzwDILP8=; b=bcbC7k8pol07TGi2Zq6kgTNF6P2X5P72t6Y5S5tDOI2PqAHl7iE1wo2HwFsauNM7dOdkpu cI0/YVvVKf6m+eDA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id uI2iEnSowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:32 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 05/34] mm, slub: remove redundant unfreeze_partials() from put_cpu_partial() Date: Wed, 9 Jun 2021 13:38:34 +0200 Message-Id: <20210609113903.1421-6-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=RJCMgcjX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=bcbC7k8p; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=RJCMgcjX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=bcbC7k8p; dmarc=none; spf=pass (imf14.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: 3qur91a9feudzkq8hga1ubyx5rcyw6hm X-Rspamd-Queue-Id: 5C6CCC00CBF2 X-HE-Tag: 1623238769-279955 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Commit d6e0b7fa1186 ("slub: make dead caches discard free slabs immediately") introduced cpu partial flushing for kmemcg caches, based on setting the target cpu_partial to 0 and adding a flushing check in put_cpu_partial(). This code that sets cpu_partial to 0 was later moved by c9fc586403e7 ("slab: introduce __kmemcg_cache_deactivate()") and ultimately removed by 9855609bde03 ("mm: memcg/slab: use a single set of kmem_caches for all accounted allocations"). However the check and flush in put_cpu_partial() was never removed, although it's effectively a dead code. So this patch removes it. Note that d6e0b7fa1186 also added preempt_disable()/enable() to unfreeze_partials() which could be thus also considered unnecessary. But further patches will rely on it, so keep it. Signed-off-by: Vlastimil Babka --- mm/slub.c | 7 ------- 1 file changed, 7 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 2953f6e43cae..f740598696b4 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2420,13 +2420,6 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) } while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page) != oldpage); - if (unlikely(!slub_cpu_partial(s))) { - unsigned long flags; - - local_irq_save(flags); - unfreeze_partials(s, this_cpu_ptr(s->cpu_slab)); - local_irq_restore(flags); - } preempt_enable(); #endif /* CONFIG_SLUB_CPU_PARTIAL */ } From patchwork Wed Jun 9 11:38:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE834C48BDF for ; Wed, 9 Jun 2021 11:39:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 802A3613B1 for ; Wed, 9 Jun 2021 11:39:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 802A3613B1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F11D6B0073; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 441846B0080; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3EE66B007E; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0189.hostedemail.com [216.40.44.189]) by kanga.kvack.org (Postfix) with ESMTP id 8E91E6B0075 for ; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 30BF2A8F8 for ; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) X-FDA: 78233990268.12.09EB7C5 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf21.hostedemail.com (Postfix) with ESMTP id 59C7EE000264 for ; Wed, 9 Jun 2021 11:39:28 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id AEEC61FD5E; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=je5OLYb88ZghiDt2tfqoGBAzw5mIHL6ZuL1ieKZAWls=; b=SP7R3Wdq+ca70GBPjuBb5lOmMYbVxAmtP/2BMfEnst17ZsSwG1e6u3JFIDQ4Jd76B3lJ9n 6c4Sq/RhqmA0sjuYKrOOt9lr3vohntoTLh3FMT/hcjBJLV/JamglaPGSlnGxD29E2EReyE eLX3CbnpmrTD5XGXo0Ethhfyz6shU30= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=je5OLYb88ZghiDt2tfqoGBAzw5mIHL6ZuL1ieKZAWls=; b=Fkqzad5/LFNHgBBJdrbJ+r7Y7W6sT9R9QS2oZCag5ntHmNxFpWdJ6ZeYV3M+1WwBwU3c2a ooqF1TGx9uDvWBAA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 7CCD6118DD; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=je5OLYb88ZghiDt2tfqoGBAzw5mIHL6ZuL1ieKZAWls=; b=SP7R3Wdq+ca70GBPjuBb5lOmMYbVxAmtP/2BMfEnst17ZsSwG1e6u3JFIDQ4Jd76B3lJ9n 6c4Sq/RhqmA0sjuYKrOOt9lr3vohntoTLh3FMT/hcjBJLV/JamglaPGSlnGxD29E2EReyE eLX3CbnpmrTD5XGXo0Ethhfyz6shU30= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=je5OLYb88ZghiDt2tfqoGBAzw5mIHL6ZuL1ieKZAWls=; b=Fkqzad5/LFNHgBBJdrbJ+r7Y7W6sT9R9QS2oZCag5ntHmNxFpWdJ6ZeYV3M+1WwBwU3c2a ooqF1TGx9uDvWBAA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id kCH+HXSowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:32 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 06/34] mm, slub: unify cmpxchg_double_slab() and __cmpxchg_double_slab() Date: Wed, 9 Jun 2021 13:38:35 +0200 Message-Id: <20210609113903.1421-7-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=SP7R3Wdq; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="Fkqzad5/"; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=SP7R3Wdq; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="Fkqzad5/"; spf=pass (imf21.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 59C7EE000264 X-Stat-Signature: trykxh1osmcosn78515p9cjydp14fkri X-HE-Tag: 1623238768-761035 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These functions differ only in irq disabling in the slow path. We can create a common function with an extra bool parameter to control the irq disabling. As the functions are inline and the parameter compile-time constant, there will be no runtime overhead due to this change. Also change the DEBUG_VM based irqs disable assert to the more standard lockdep_assert based one. Signed-off-by: Vlastimil Babka --- mm/slub.c | 62 +++++++++++++++++++++---------------------------------- 1 file changed, 24 insertions(+), 38 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index f740598696b4..76af5065baeb 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -365,13 +365,13 @@ static __always_inline void slab_unlock(struct page *page) __bit_spin_unlock(PG_locked, &page->flags); } -/* Interrupts must be disabled (for the fallback code to work right) */ -static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct page *page, +static inline bool ___cmpxchg_double_slab(struct kmem_cache *s, struct page *page, void *freelist_old, unsigned long counters_old, void *freelist_new, unsigned long counters_new, - const char *n) + const char *n, bool disable_irqs) { - VM_BUG_ON(!irqs_disabled()); + if (!disable_irqs) + lockdep_assert_irqs_disabled(); #if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) if (s->flags & __CMPXCHG_DOUBLE) { @@ -382,15 +382,23 @@ static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct page *page } else #endif { + unsigned long flags; + + if (disable_irqs) + local_irq_save(flags); slab_lock(page); if (page->freelist == freelist_old && page->counters == counters_old) { page->freelist = freelist_new; page->counters = counters_new; slab_unlock(page); + if (disable_irqs) + local_irq_restore(flags); return true; } slab_unlock(page); + if (disable_irqs) + local_irq_restore(flags); } cpu_relax(); @@ -403,45 +411,23 @@ static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct page *page return false; } -static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page, +/* Interrupts must be disabled (for the fallback code to work right) */ +static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct page *page, void *freelist_old, unsigned long counters_old, void *freelist_new, unsigned long counters_new, const char *n) { -#if defined(CONFIG_HAVE_CMPXCHG_DOUBLE) && \ - defined(CONFIG_HAVE_ALIGNED_STRUCT_PAGE) - if (s->flags & __CMPXCHG_DOUBLE) { - if (cmpxchg_double(&page->freelist, &page->counters, - freelist_old, counters_old, - freelist_new, counters_new)) - return true; - } else -#endif - { - unsigned long flags; - - local_irq_save(flags); - slab_lock(page); - if (page->freelist == freelist_old && - page->counters == counters_old) { - page->freelist = freelist_new; - page->counters = counters_new; - slab_unlock(page); - local_irq_restore(flags); - return true; - } - slab_unlock(page); - local_irq_restore(flags); - } - - cpu_relax(); - stat(s, CMPXCHG_DOUBLE_FAIL); - -#ifdef SLUB_DEBUG_CMPXCHG - pr_info("%s %s: cmpxchg double redo ", n, s->name); -#endif + return ___cmpxchg_double_slab(s, page, freelist_old, counters_old, + freelist_new, counters_new, n, false); +} - return false; +static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page, + void *freelist_old, unsigned long counters_old, + void *freelist_new, unsigned long counters_new, + const char *n) +{ + return ___cmpxchg_double_slab(s, page, freelist_old, counters_old, + freelist_new, counters_new, n, true); } #ifdef CONFIG_SLUB_DEBUG From patchwork Wed Jun 9 11:38:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFBBBC48BCD for ; Wed, 9 Jun 2021 11:39:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 84CBA613B9 for ; Wed, 9 Jun 2021 11:39:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 84CBA613B9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9E4256B0075; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 662896B0074; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEDE36B0075; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0210.hostedemail.com [216.40.44.210]) by kanga.kvack.org (Postfix) with ESMTP id B0A366B007D for ; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5932682499B9 for ; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) X-FDA: 78233990268.25.596499E Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf27.hostedemail.com (Postfix) with ESMTP id A2F4D8019358 for ; Wed, 9 Jun 2021 11:39:29 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E00901FD60; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kZIe51L2fie5iShDaR2U3UI0fOHDEXk4ChagcFC2+ro=; b=A0yIMrqov5NmBBLi2AVJ1tlUhlrOyDcdzLY1vRFQheIZRzdtdAWInV53TmmjbnlZ04RAuW zjBeRes0Ztz/i7lQK0Ric8dILMzgkhIJ6cWFSVV0DMQ1mc8IUZBF1JH48EGb/hSTtynmvr TatvHLfnk224Ac1TOJYY+KtAvBwZ4qE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kZIe51L2fie5iShDaR2U3UI0fOHDEXk4ChagcFC2+ro=; b=JCwf9VUextUA6e0GYf8JV5DyQYu6ST17sVIbtjo9s92vcD0kDwZmiKQ7WccNf1l2lMUh0P 4dG7PTee1KjZfgCw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id B174A11A98; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kZIe51L2fie5iShDaR2U3UI0fOHDEXk4ChagcFC2+ro=; b=A0yIMrqov5NmBBLi2AVJ1tlUhlrOyDcdzLY1vRFQheIZRzdtdAWInV53TmmjbnlZ04RAuW zjBeRes0Ztz/i7lQK0Ric8dILMzgkhIJ6cWFSVV0DMQ1mc8IUZBF1JH48EGb/hSTtynmvr TatvHLfnk224Ac1TOJYY+KtAvBwZ4qE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238772; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kZIe51L2fie5iShDaR2U3UI0fOHDEXk4ChagcFC2+ro=; b=JCwf9VUextUA6e0GYf8JV5DyQYu6ST17sVIbtjo9s92vcD0kDwZmiKQ7WccNf1l2lMUh0P 4dG7PTee1KjZfgCw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id 8PjIKnSowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:32 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 07/34] mm, slub: extract get_partial() from new_slab_objects() Date: Wed, 9 Jun 2021 13:38:36 +0200 Message-Id: <20210609113903.1421-8-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A2F4D8019358 Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=A0yIMrqo; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=JCwf9VUe; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=A0yIMrqo; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=JCwf9VUe; spf=pass (imf27.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: g9uzpcse7amt1p6k3gt1abe5jp1uj6f9 X-HE-Tag: 1623238769-700702 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The later patches will need more fine grained control over individual actions in ___slab_alloc(), the only caller of new_slab_objects(), so this is a first preparatory step with no functional change. This adds a goto label that appears unnecessary at this point, but will be useful for later changes. Signed-off-by: Vlastimil Babka Acked-by: Christoph Lameter --- mm/slub.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 76af5065baeb..38256e87dde8 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2553,17 +2553,12 @@ slab_out_of_memory(struct kmem_cache *s, gfp_t gfpflags, int nid) static inline void *new_slab_objects(struct kmem_cache *s, gfp_t flags, int node, struct kmem_cache_cpu **pc) { - void *freelist; + void *freelist = NULL; struct kmem_cache_cpu *c = *pc; struct page *page; WARN_ON_ONCE(s->ctor && (flags & __GFP_ZERO)); - freelist = get_partial(s, flags, node, c); - - if (freelist) - return freelist; - page = new_slab(s, flags, node); if (page) { c = raw_cpu_ptr(s->cpu_slab); @@ -2727,6 +2722,10 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto redo; } + freelist = get_partial(s, gfpflags, node, c); + if (freelist) + goto check_new_page; + freelist = new_slab_objects(s, gfpflags, node, &c); if (unlikely(!freelist)) { @@ -2734,6 +2733,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, return NULL; } +check_new_page: page = c->page; if (likely(!kmem_cache_debug(s) && pfmemalloc_match(page, gfpflags))) goto load_freelist; From patchwork Wed Jun 9 11:38:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0504C48BDF for ; Wed, 9 Jun 2021 11:39:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 660D061364 for ; Wed, 9 Jun 2021 11:39:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 660D061364 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BFD9D6B0074; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 951366B0081; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 466E16B007B; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0202.hostedemail.com [216.40.44.202]) by kanga.kvack.org (Postfix) with ESMTP id 033156B0073 for ; Wed, 9 Jun 2021 07:39:34 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9EC8081F3 for ; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) X-FDA: 78233990268.28.14BA918 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf10.hostedemail.com (Postfix) with ESMTP id B277E4202A19 for ; Wed, 9 Jun 2021 11:39:30 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 1E8681FD61; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MP6gY1FO49owhq4IXmCsqEM0wX1JIeYz2iR8RuLeKzs=; b=V8FHGEI4P8WxPddKk/LZH2BMfi9s9m7qRpZAJLGT2feGyXEA8FTdsmKjBZNWadQxLFQUiT V/ZP0clOdhvk/IjoVdzJ44DcjpMaZtZ+xJ7azBjv5jXg7BrzfVICwpeKmOLDeBRoxMDMsK Mrv3WVt6qvFCdN3M7N25U/n6Qzu6NiA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MP6gY1FO49owhq4IXmCsqEM0wX1JIeYz2iR8RuLeKzs=; b=gRRa2lgyuO6eUIZ/b/bgZn4ZAf2Jx9qQ7KZvvitRhlT7Qi+oCsDSg6bDJ1kQOHQyvVw8c3 t5cKvFrVti1+r6Dg== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id E2588118DD; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MP6gY1FO49owhq4IXmCsqEM0wX1JIeYz2iR8RuLeKzs=; b=V8FHGEI4P8WxPddKk/LZH2BMfi9s9m7qRpZAJLGT2feGyXEA8FTdsmKjBZNWadQxLFQUiT V/ZP0clOdhvk/IjoVdzJ44DcjpMaZtZ+xJ7azBjv5jXg7BrzfVICwpeKmOLDeBRoxMDMsK Mrv3WVt6qvFCdN3M7N25U/n6Qzu6NiA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MP6gY1FO49owhq4IXmCsqEM0wX1JIeYz2iR8RuLeKzs=; b=gRRa2lgyuO6eUIZ/b/bgZn4ZAf2Jx9qQ7KZvvitRhlT7Qi+oCsDSg6bDJ1kQOHQyvVw8c3 t5cKvFrVti1+r6Dg== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id 4EvANnSowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:32 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 08/34] mm, slub: dissolve new_slab_objects() into ___slab_alloc() Date: Wed, 9 Jun 2021 13:38:37 +0200 Message-Id: <20210609113903.1421-9-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: B277E4202A19 X-Stat-Signature: 5aq7xxq61kzy68nekbosaenhxp5ciwpm Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=V8FHGEI4; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=gRRa2lgy; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=V8FHGEI4; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=gRRa2lgy; dmarc=none; spf=pass (imf10.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-HE-Tag: 1623238770-182832 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The later patches will need more fine grained control over individual actions in ___slab_alloc(), the only caller of new_slab_objects(), so dissolve it there. This is a preparatory step with no functional change. The only minor change is moving WARN_ON_ONCE() for using a constructor together with __GFP_ZERO to new_slab(), which makes it somewhat less frequent, but still able to catch a development change introducing a systematic misuse. Signed-off-by: Vlastimil Babka Acked-by: Christoph Lameter Acked-by: Mel Gorman --- mm/slub.c | 50 ++++++++++++++++++-------------------------------- 1 file changed, 18 insertions(+), 32 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 38256e87dde8..87f652bf8e4d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1825,6 +1825,8 @@ static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int node) if (unlikely(flags & GFP_SLAB_BUG_MASK)) flags = kmalloc_fix_flags(flags); + WARN_ON_ONCE(s->ctor && (flags & __GFP_ZERO)); + return allocate_slab(s, flags & (GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK), node); } @@ -2550,36 +2552,6 @@ slab_out_of_memory(struct kmem_cache *s, gfp_t gfpflags, int nid) #endif } -static inline void *new_slab_objects(struct kmem_cache *s, gfp_t flags, - int node, struct kmem_cache_cpu **pc) -{ - void *freelist = NULL; - struct kmem_cache_cpu *c = *pc; - struct page *page; - - WARN_ON_ONCE(s->ctor && (flags & __GFP_ZERO)); - - page = new_slab(s, flags, node); - if (page) { - c = raw_cpu_ptr(s->cpu_slab); - if (c->page) - flush_slab(s, c); - - /* - * No other reference to the page yet so we can - * muck around with it freely without cmpxchg - */ - freelist = page->freelist; - page->freelist = NULL; - - stat(s, ALLOC_SLAB); - c->page = page; - *pc = c; - } - - return freelist; -} - static inline bool pfmemalloc_match(struct page *page, gfp_t gfpflags) { if (unlikely(PageSlabPfmemalloc(page))) @@ -2726,13 +2698,27 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (freelist) goto check_new_page; - freelist = new_slab_objects(s, gfpflags, node, &c); + page = new_slab(s, gfpflags, node); - if (unlikely(!freelist)) { + if (unlikely(!page)) { slab_out_of_memory(s, gfpflags, node); return NULL; } + c = raw_cpu_ptr(s->cpu_slab); + if (c->page) + flush_slab(s, c); + + /* + * No other reference to the page yet so we can + * muck around with it freely without cmpxchg + */ + freelist = page->freelist; + page->freelist = NULL; + + stat(s, ALLOC_SLAB); + c->page = page; + check_new_page: page = c->page; if (likely(!kmem_cache_debug(s) && pfmemalloc_match(page, gfpflags))) From patchwork Wed Jun 9 11:38:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AC0FC48BCD for ; Wed, 9 Jun 2021 11:39:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0926760FD8 for ; Wed, 9 Jun 2021 11:39:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0926760FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 460F86B007D; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A4556B0087; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EEF126B0082; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0179.hostedemail.com [216.40.44.179]) by kanga.kvack.org (Postfix) with ESMTP id 86DE26B007D for ; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2FCCA180AD81A for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) X-FDA: 78233990310.11.6DFA693 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf26.hostedemail.com (Postfix) with ESMTP id 671C34202A07 for ; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 4E7121FD62; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VxG0sTKG3HDQvTVC+wgypTAYjlpWe1E+zR9CMgBlKVE=; b=VgaBv0jv8ekGLTGkPcWtSbBkK43e1mKqGkYzPFDwmnaQICdiJz28h5LXVJ/Z8epVWA2xxk 3kCpd0oPsN6tLFiWd841pTIPXwPtECiyX9oSrEJHb88boahSQCiiMRQbC+RERHzAyOCw0F vdlM7+8myOV7dhM+cf+dskR830UQQ6U= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VxG0sTKG3HDQvTVC+wgypTAYjlpWe1E+zR9CMgBlKVE=; b=siZMeoMpA+713vHHCpAmN4iXON34m+uYDBrbCgJ2wjCc+KWDxs0r8wr3P/59DWT3a9JDCe 87KmRma2rVo19WBA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 2010D11A98; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VxG0sTKG3HDQvTVC+wgypTAYjlpWe1E+zR9CMgBlKVE=; b=VgaBv0jv8ekGLTGkPcWtSbBkK43e1mKqGkYzPFDwmnaQICdiJz28h5LXVJ/Z8epVWA2xxk 3kCpd0oPsN6tLFiWd841pTIPXwPtECiyX9oSrEJHb88boahSQCiiMRQbC+RERHzAyOCw0F vdlM7+8myOV7dhM+cf+dskR830UQQ6U= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VxG0sTKG3HDQvTVC+wgypTAYjlpWe1E+zR9CMgBlKVE=; b=siZMeoMpA+713vHHCpAmN4iXON34m+uYDBrbCgJ2wjCc+KWDxs0r8wr3P/59DWT3a9JDCe 87KmRma2rVo19WBA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id QOlbB3WowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:33 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 09/34] mm, slub: return slab page from get_partial() and set c->page afterwards Date: Wed, 9 Jun 2021 13:38:38 +0200 Message-Id: <20210609113903.1421-10-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 671C34202A07 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=VgaBv0jv; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=siZMeoMp; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=VgaBv0jv; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=siZMeoMp; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: jg566xceyhz9k793fxamwjpg1qzxqi6p X-HE-Tag: 1623238772-47880 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The function get_partial() finds a suitable page on a partial list, acquires and returns its freelist and assigns the page pointer to kmem_cache_cpu. In later patch we will need more control over the kmem_cache_cpu.page assignment, so instead of passing a kmem_cache_cpu pointer, pass a pointer to a pointer to a page that get_partial() can fill and the caller can assign the kmem_cache_cpu.page pointer. No functional change as all of this still happens with disabled IRQs. Signed-off-by: Vlastimil Babka --- mm/slub.c | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 87f652bf8e4d..ca7cfc7706cc 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1957,7 +1957,7 @@ static inline bool pfmemalloc_match(struct page *page, gfp_t gfpflags); * Try to allocate a partial slab from a specific node. */ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, - struct kmem_cache_cpu *c, gfp_t flags) + struct page **ret_page, gfp_t flags) { struct page *page, *page2; void *object = NULL; @@ -1986,7 +1986,7 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, available += objects; if (!object) { - c->page = page; + *ret_page = page; stat(s, ALLOC_FROM_PARTIAL); object = t; } else { @@ -2006,7 +2006,7 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, * Get a page from somewhere. Search in increasing NUMA distances. */ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, - struct kmem_cache_cpu *c) + struct page **ret_page) { #ifdef CONFIG_NUMA struct zonelist *zonelist; @@ -2048,7 +2048,7 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, if (n && cpuset_zone_allowed(zone, flags) && n->nr_partial > s->min_partial) { - object = get_partial_node(s, n, c, flags); + object = get_partial_node(s, n, ret_page, flags); if (object) { /* * Don't check read_mems_allowed_retry() @@ -2070,7 +2070,7 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, * Get a partial page, lock it and return it. */ static void *get_partial(struct kmem_cache *s, gfp_t flags, int node, - struct kmem_cache_cpu *c) + struct page **ret_page) { void *object; int searchnode = node; @@ -2078,11 +2078,11 @@ static void *get_partial(struct kmem_cache *s, gfp_t flags, int node, if (node == NUMA_NO_NODE) searchnode = numa_mem_id(); - object = get_partial_node(s, get_node(s, searchnode), c, flags); + object = get_partial_node(s, get_node(s, searchnode), ret_page, flags); if (object || node != NUMA_NO_NODE) return object; - return get_any_partial(s, flags, c); + return get_any_partial(s, flags, ret_page); } #ifdef CONFIG_PREEMPTION @@ -2694,9 +2694,11 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto redo; } - freelist = get_partial(s, gfpflags, node, c); - if (freelist) + freelist = get_partial(s, gfpflags, node, &page); + if (freelist) { + c->page = page; goto check_new_page; + } page = new_slab(s, gfpflags, node); @@ -2720,7 +2722,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, c->page = page; check_new_page: - page = c->page; if (likely(!kmem_cache_debug(s) && pfmemalloc_match(page, gfpflags))) goto load_freelist; From patchwork Wed Jun 9 11:38:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32646C48BD1 for ; Wed, 9 Jun 2021 11:39:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D306D60FD8 for ; Wed, 9 Jun 2021 11:39:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D306D60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F6FC6B007B; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 665506B008A; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0480D6B0081; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0096.hostedemail.com [216.40.44.96]) by kanga.kvack.org (Postfix) with ESMTP id A455C6B007B for ; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 23573BA10 for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) X-FDA: 78233990310.15.5F9FB61 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf29.hostedemail.com (Postfix) with ESMTP id D866D548 for ; Wed, 9 Jun 2021 11:39:28 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7F3091FD64; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YZTiZ2krokRZPz74ttwlqY3E9OPfJw2MVLDApZp5kJ0=; b=LDp5gkLsDT1plakKx1QbWTNmHjfTbVB/ZEIRrspp9/GdglbQNNmsfwGjZkEpn/Sxy7U7/U zy0rjY96XUhFuNvSNBUY45+z6MUydML6SOyuq1HAgqSj1C7hvXRl13bX+p27x0FPiHEk72 drEpQmA9Kgi6O5+zbkdunga9lqCMy3A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YZTiZ2krokRZPz74ttwlqY3E9OPfJw2MVLDApZp5kJ0=; b=z65b+i1Nn4+r+AFVNc7LKQ3aYAOq7ZlkVGPjGctN+mrDVuLJ9vgCJ260MIpvYEty25LiyI O/d/ETQxIENCwTCw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 507FE118DD; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YZTiZ2krokRZPz74ttwlqY3E9OPfJw2MVLDApZp5kJ0=; b=LDp5gkLsDT1plakKx1QbWTNmHjfTbVB/ZEIRrspp9/GdglbQNNmsfwGjZkEpn/Sxy7U7/U zy0rjY96XUhFuNvSNBUY45+z6MUydML6SOyuq1HAgqSj1C7hvXRl13bX+p27x0FPiHEk72 drEpQmA9Kgi6O5+zbkdunga9lqCMy3A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YZTiZ2krokRZPz74ttwlqY3E9OPfJw2MVLDApZp5kJ0=; b=z65b+i1Nn4+r+AFVNc7LKQ3aYAOq7ZlkVGPjGctN+mrDVuLJ9vgCJ260MIpvYEty25LiyI O/d/ETQxIENCwTCw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id OPssE3WowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:33 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 10/34] mm, slub: restructure new page checks in ___slab_alloc() Date: Wed, 9 Jun 2021 13:38:39 +0200 Message-Id: <20210609113903.1421-11-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=LDp5gkLs; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=z65b+i1N; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=LDp5gkLs; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=z65b+i1N; dmarc=none; spf=pass (imf29.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: xrznmbetg3cnkpdsr3s1opegwcg48izg X-Rspamd-Queue-Id: D866D548 X-HE-Tag: 1623238768-683446 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When we allocate slab object from a newly acquired page (from node's partial list or page allocator), we usually also retain the page as a new percpu slab. There are two exceptions - when pfmemalloc status of the page doesn't match our gfp flags, or when the cache has debugging enabled. The current code for these decisions is not easy to follow, so restructure it and add comments. The new structure will also help with the following changes. No functional change. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman --- mm/slub.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index ca7cfc7706cc..0d5c253a39b2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2722,13 +2722,29 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, c->page = page; check_new_page: - if (likely(!kmem_cache_debug(s) && pfmemalloc_match(page, gfpflags))) - goto load_freelist; - /* Only entered in the debug case */ - if (kmem_cache_debug(s) && - !alloc_debug_processing(s, page, freelist, addr)) - goto new_slab; /* Slab failed checks. Next slab needed */ + if (kmem_cache_debug(s)) { + if (!alloc_debug_processing(s, page, freelist, addr)) + /* Slab failed checks. Next slab needed */ + goto new_slab; + else + /* + * For debug case, we don't load freelist so that all + * allocations go through alloc_debug_processing() + */ + goto return_single; + } + + if (unlikely(!pfmemalloc_match(page, gfpflags))) + /* + * For !pfmemalloc_match() case we don't load freelist so that + * we don't make further mismatched allocations easier. + */ + goto return_single; + + goto load_freelist; + +return_single: deactivate_slab(s, page, get_freepointer(s, freelist), c); return freelist; From patchwork Wed Jun 9 11:38:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F732C48BCF for ; Wed, 9 Jun 2021 11:39:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4398D60FD8 for ; Wed, 9 Jun 2021 11:39:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4398D60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 136036B0080; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0982F6B007D; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB77C6B0083; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0200.hostedemail.com [216.40.44.200]) by kanga.kvack.org (Postfix) with ESMTP id 8837D6B007E for ; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 31DCBBBC5 for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) X-FDA: 78233990310.25.E45A2A3 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf05.hostedemail.com (Postfix) with ESMTP id AED2AE000251 for ; Wed, 9 Jun 2021 11:39:29 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id B0C2F1FD65; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SlrL7OICvBFstNZZ2XJMriwG3NzNdu907jHbbChxuc0=; b=CjpVp1rLvDZ8dbeBqHACPad2h/djS+XV2r2Cj3do2Z8zJCcvNb2LUc7OaGw7nz8jSBMrce hPm05vums1nTQYrZuX3cyHPuk6MrLNu6uD7Yi8AgBjy1UvKVT2U6fSCoU70yTi2wbXXnJ1 LQn1zYDMtQEE95DVYNivcrhGabQyBkU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SlrL7OICvBFstNZZ2XJMriwG3NzNdu907jHbbChxuc0=; b=chweC4mL0L1KS071i9+/f0T+CnbFaP0Q3h2vvFwS8OMkyyyELVt2u99fphybzOc8OHFKfP if05UsHtBxkDzjBQ== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 8155C11A98; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SlrL7OICvBFstNZZ2XJMriwG3NzNdu907jHbbChxuc0=; b=CjpVp1rLvDZ8dbeBqHACPad2h/djS+XV2r2Cj3do2Z8zJCcvNb2LUc7OaGw7nz8jSBMrce hPm05vums1nTQYrZuX3cyHPuk6MrLNu6uD7Yi8AgBjy1UvKVT2U6fSCoU70yTi2wbXXnJ1 LQn1zYDMtQEE95DVYNivcrhGabQyBkU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SlrL7OICvBFstNZZ2XJMriwG3NzNdu907jHbbChxuc0=; b=chweC4mL0L1KS071i9+/f0T+CnbFaP0Q3h2vvFwS8OMkyyyELVt2u99fphybzOc8OHFKfP if05UsHtBxkDzjBQ== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id OLEaH3WowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:33 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 11/34] mm, slub: simplify kmem_cache_cpu and tid setup Date: Wed, 9 Jun 2021 13:38:40 +0200 Message-Id: <20210609113903.1421-12-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=CjpVp1rL; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=chweC4mL; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=CjpVp1rL; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=chweC4mL; spf=pass (imf05.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: AED2AE000251 X-Stat-Signature: pgs5snhabtgjmjpu6wg886tdocuz5hc5 X-HE-Tag: 1623238769-937362 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In slab_alloc_node() and do_slab_free() fastpaths we need to guarantee that our kmem_cache_cpu pointer is from the same cpu as the tid value. Currently that's done by reading the tid first using this_cpu_read(), then the kmem_cache_cpu pointer and verifying we read the same tid using the pointer and plain READ_ONCE(). This can be simplified to just fetching kmem_cache_cpu pointer and then reading tid using the pointer. That guarantees they are from the same cpu. We don't need to read the tid using this_cpu_read() because the value will be validated by this_cpu_cmpxchg_double(), making sure we are on the correct cpu and the freelist didn't change by anyone preempting us since reading the tid. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman --- mm/slub.c | 22 +++++++++------------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 0d5c253a39b2..6d6a9a69db8a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2822,15 +2822,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, * reading from one cpu area. That does not matter as long * as we end up on the original cpu again when doing the cmpxchg. * - * We should guarantee that tid and kmem_cache are retrieved on - * the same cpu. It could be different if CONFIG_PREEMPTION so we need - * to check if it is matched or not. + * We must guarantee that tid and kmem_cache_cpu are retrieved on the + * same cpu. We read first the kmem_cache_cpu pointer and use it to read + * the tid. If we are preempted and switched to another cpu between the + * two reads, it's OK as the two are still associated with the same cpu + * and cmpxchg later will validate the cpu. */ - do { - tid = this_cpu_read(s->cpu_slab->tid); - c = raw_cpu_ptr(s->cpu_slab); - } while (IS_ENABLED(CONFIG_PREEMPTION) && - unlikely(tid != READ_ONCE(c->tid))); + c = raw_cpu_ptr(s->cpu_slab); + tid = READ_ONCE(c->tid); /* * Irqless object alloc/free algorithm used here depends on sequence @@ -3104,11 +3103,8 @@ static __always_inline void do_slab_free(struct kmem_cache *s, * data is retrieved via this pointer. If we are on the same cpu * during the cmpxchg then the free will succeed. */ - do { - tid = this_cpu_read(s->cpu_slab->tid); - c = raw_cpu_ptr(s->cpu_slab); - } while (IS_ENABLED(CONFIG_PREEMPTION) && - unlikely(tid != READ_ONCE(c->tid))); + c = raw_cpu_ptr(s->cpu_slab); + tid = READ_ONCE(c->tid); /* Same with comment on barrier() in slab_alloc_node() */ barrier(); From patchwork Wed Jun 9 11:38:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D08EEC48BCD for ; Wed, 9 Jun 2021 11:39:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 80B0360FD8 for ; Wed, 9 Jun 2021 11:39:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 80B0360FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A10C36B0085; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E9246B0081; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 132E86B0085; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0227.hostedemail.com [216.40.44.227]) by kanga.kvack.org (Postfix) with ESMTP id B6B426B0080 for ; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 4B88B180AD81D for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) X-FDA: 78233990310.13.307521F Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf11.hostedemail.com (Postfix) with ESMTP id 7468A200107C for ; Wed, 9 Jun 2021 11:39:30 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E4CDC1FD66; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ooHJLHsolbs8E/VsJWSGGh6r7etfcAEkF/VGN97pHs8=; b=2FEH2m+jM7t9BeLHL/Td6f3/iPF6vDWFE8O+HEgdPnkbkLdupKHuf3iMD3WRfepF1/SXou 78IPycdJLQ6jBbmv0rwGpm5Ruv4I3Fw76WkDXDaxMBeYxrrM3GpoXKVKaHutEws0EbRdk1 FE04A1MD2LKKfljjGhEVVuJjTdawPpU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ooHJLHsolbs8E/VsJWSGGh6r7etfcAEkF/VGN97pHs8=; b=LB74xd0RZSX+bW5kgk90zniYjKzdgnWOXfaverf0Zf5WSXwCuUKZ39fe2hWw1GMto0X9Ja R9014SoOeFha/JBA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id B2714118DD; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ooHJLHsolbs8E/VsJWSGGh6r7etfcAEkF/VGN97pHs8=; b=2FEH2m+jM7t9BeLHL/Td6f3/iPF6vDWFE8O+HEgdPnkbkLdupKHuf3iMD3WRfepF1/SXou 78IPycdJLQ6jBbmv0rwGpm5Ruv4I3Fw76WkDXDaxMBeYxrrM3GpoXKVKaHutEws0EbRdk1 FE04A1MD2LKKfljjGhEVVuJjTdawPpU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238773; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ooHJLHsolbs8E/VsJWSGGh6r7etfcAEkF/VGN97pHs8=; b=LB74xd0RZSX+bW5kgk90zniYjKzdgnWOXfaverf0Zf5WSXwCuUKZ39fe2hWw1GMto0X9Ja R9014SoOeFha/JBA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id mCERK3WowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:33 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 12/34] mm, slub: move disabling/enabling irqs to ___slab_alloc() Date: Wed, 9 Jun 2021 13:38:41 +0200 Message-Id: <20210609113903.1421-13-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 7468A200107C X-Stat-Signature: 8zahtgx49gxsmbnnadfg69xekdarxjfn Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=2FEH2m+j; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=LB74xd0R; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=2FEH2m+j; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=LB74xd0R; dmarc=none; spf=pass (imf11.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-HE-Tag: 1623238770-271504 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently __slab_alloc() disables irqs around the whole ___slab_alloc(). This includes cases where this is not needed, such as when the allocation ends up in the page allocator and has to awkwardly enable irqs back based on gfp flags. Also the whole kmem_cache_alloc_bulk() is executed with irqs disabled even when it hits the __slab_alloc() slow path, and long periods with disabled interrupts are undesirable. As a first step towards reducing irq disabled periods, move irq handling into ___slab_alloc(). Callers will instead prevent the s->cpu_slab percpu pointer from becoming invalid via get_cpu_ptr(), thus preempt_disable(). This does not protect against modification by an irq handler, which is still done by disabled irq for most of ___slab_alloc(). As a small immediate benefit, slab_out_of_memory() from ___slab_alloc() is now called with irqs enabled. kmem_cache_alloc_bulk() disables irqs for its fastpath and then re-enables them before calling ___slab_alloc(), which then disables them at its discretion. The whole kmem_cache_alloc_bulk() operation also disables preemption. When ___slab_alloc() calls new_slab() to allocate a new page, re-enable preemption, because new_slab() will re-enable interrupts in contexts that allow blocking (this will be improved by later patches). The patch itself will thus increase overhead a bit due to disabled preemption and increased disabling/enabling irqs in kmem_cache_alloc_bulk(), but that will be gradually improved in the following patches. Signed-off-by: Vlastimil Babka --- mm/slub.c | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 6d6a9a69db8a..4800d768d0d3 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2610,7 +2610,7 @@ static inline void *get_freelist(struct kmem_cache *s, struct page *page) * we need to allocate a new slab. This is the slowest path since it involves * a call to the page allocator and the setup of a new slab. * - * Version of __slab_alloc to use when we know that interrupts are + * Version of __slab_alloc to use when we know that preemption is * already disabled (which is the case for bulk allocation). */ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, @@ -2618,9 +2618,11 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, { void *freelist; struct page *page; + unsigned long flags; stat(s, ALLOC_SLOWPATH); + local_irq_save(flags); page = c->page; if (!page) { /* @@ -2683,6 +2685,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, VM_BUG_ON(!c->page->frozen); c->freelist = get_freepointer(s, freelist); c->tid = next_tid(c->tid); + local_irq_restore(flags); return freelist; new_slab: @@ -2700,14 +2703,16 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto check_new_page; } + put_cpu_ptr(s->cpu_slab); page = new_slab(s, gfpflags, node); + c = get_cpu_ptr(s->cpu_slab); if (unlikely(!page)) { + local_irq_restore(flags); slab_out_of_memory(s, gfpflags, node); return NULL; } - c = raw_cpu_ptr(s->cpu_slab); if (c->page) flush_slab(s, c); @@ -2747,31 +2752,33 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, return_single: deactivate_slab(s, page, get_freepointer(s, freelist), c); + local_irq_restore(flags); return freelist; } /* - * Another one that disabled interrupt and compensates for possible - * cpu changes by refetching the per cpu area pointer. + * A wrapper for ___slab_alloc() for contexts where preemption is not yet + * disabled. Compensates for possible cpu changes by refetching the per cpu area + * pointer. */ static void *__slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, unsigned long addr, struct kmem_cache_cpu *c) { void *p; - unsigned long flags; - local_irq_save(flags); #ifdef CONFIG_PREEMPTION /* * We may have been preempted and rescheduled on a different - * cpu before disabling interrupts. Need to reload cpu area + * cpu before disabling preemption. Need to reload cpu area * pointer. */ - c = this_cpu_ptr(s->cpu_slab); + c = get_cpu_ptr(s->cpu_slab); #endif p = ___slab_alloc(s, gfpflags, node, addr, c); - local_irq_restore(flags); +#ifdef CONFIG_PREEMPTION + put_cpu_ptr(s->cpu_slab); +#endif return p; } @@ -3291,8 +3298,8 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, * IRQs, which protects against PREEMPT and interrupts * handlers invoking normal fastpath. */ + c = get_cpu_ptr(s->cpu_slab); local_irq_disable(); - c = this_cpu_ptr(s->cpu_slab); for (i = 0; i < size; i++) { void *object = kfence_alloc(s, s->object_size, flags); @@ -3313,6 +3320,8 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, */ c->tid = next_tid(c->tid); + local_irq_enable(); + /* * Invoking slow path likely have side-effect * of re-populating per CPU c->freelist @@ -3325,6 +3334,8 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, c = this_cpu_ptr(s->cpu_slab); maybe_wipe_obj_freeptr(s, p[i]); + local_irq_disable(); + continue; /* goto for-loop */ } c->freelist = get_freepointer(s, object); @@ -3333,6 +3344,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, } c->tid = next_tid(c->tid); local_irq_enable(); + put_cpu_ptr(s->cpu_slab); /* * memcg and kmem_cache debug support and memory initialization. From patchwork Wed Jun 9 11:38:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9E3EC48BCD for ; Wed, 9 Jun 2021 11:40:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7E54D60FD8 for ; Wed, 9 Jun 2021 11:40:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7E54D60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 000E76B0083; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCDF56B008A; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8AE166B0087; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E7E746B007E for ; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 8C997180AD81F for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) X-FDA: 78233990310.36.6663209 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf20.hostedemail.com (Postfix) with ESMTP id 0CF9C3C3 for ; Wed, 9 Jun 2021 11:39:29 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 23B441FD67; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CxpLaMxa6ZmRdqzKDifZCdS7C7uCUIH4tSWBVxk980E=; b=jynYDQ9tsDdezhA842kVnKvxitXrThZiCoy+eZDBBiQDMF8uVojY4yHKvEv/LPk8cmqbZn TVJLRj5fzZ5eJoACdeSS8sCZXHWVg5RE9gp6RW11VQ74OVG/sX++y6v6cg6dg//Y6l3H8e QOXupKg12nw5rn7/yhH3W6o4wvllfRk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CxpLaMxa6ZmRdqzKDifZCdS7C7uCUIH4tSWBVxk980E=; b=GGhWiCyXSgVvskjNO5jxfCo8TtjhkArWCimS5dFBbW7eOi62CXHHHDT077tVrZcgPNIBBS 3qwvDkkfoccGniDw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id E8A2211A98; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CxpLaMxa6ZmRdqzKDifZCdS7C7uCUIH4tSWBVxk980E=; b=jynYDQ9tsDdezhA842kVnKvxitXrThZiCoy+eZDBBiQDMF8uVojY4yHKvEv/LPk8cmqbZn TVJLRj5fzZ5eJoACdeSS8sCZXHWVg5RE9gp6RW11VQ74OVG/sX++y6v6cg6dg//Y6l3H8e QOXupKg12nw5rn7/yhH3W6o4wvllfRk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CxpLaMxa6ZmRdqzKDifZCdS7C7uCUIH4tSWBVxk980E=; b=GGhWiCyXSgVvskjNO5jxfCo8TtjhkArWCimS5dFBbW7eOi62CXHHHDT077tVrZcgPNIBBS 3qwvDkkfoccGniDw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id uCfPN3WowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:33 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 13/34] mm, slub: do initial checks in ___slab_alloc() with irqs enabled Date: Wed, 9 Jun 2021 13:38:42 +0200 Message-Id: <20210609113903.1421-14-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0CF9C3C3 X-Stat-Signature: 5n7d7q6z6ayk1au8djza9i9mu8kcuygu Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=jynYDQ9t; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=GGhWiCyX; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=jynYDQ9t; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=GGhWiCyX; dmarc=none; spf=pass (imf20.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-HE-Tag: 1623238769-747520 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As another step of shortening irq disabled sections in ___slab_alloc(), delay disabling irqs until we pass the initial checks if there is a cached percpu slab and it's suitable for our allocation. Now we have to recheck c->page after actually disabling irqs as an allocation in irq handler might have replaced it. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman --- mm/slub.c | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 4800d768d0d3..22b5bfef8aa6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2622,8 +2622,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, stat(s, ALLOC_SLOWPATH); - local_irq_save(flags); - page = c->page; +reread_page: + + page = READ_ONCE(c->page); if (!page) { /* * if the node is not online or has no normal memory, just @@ -2632,6 +2633,11 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (unlikely(node != NUMA_NO_NODE && !node_isset(node, slab_nodes))) node = NUMA_NO_NODE; + local_irq_save(flags); + if (unlikely(c->page)) { + local_irq_restore(flags); + goto reread_page; + } goto new_slab; } redo: @@ -2646,8 +2652,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto redo; } else { stat(s, ALLOC_NODE_MISMATCH); - deactivate_slab(s, page, c->freelist, c); - goto new_slab; + goto deactivate_slab; } } @@ -2656,12 +2661,15 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, * PFMEMALLOC but right now, we are losing the pfmemalloc * information when the page leaves the per-cpu allocator */ - if (unlikely(!pfmemalloc_match(page, gfpflags))) { - deactivate_slab(s, page, c->freelist, c); - goto new_slab; - } + if (unlikely(!pfmemalloc_match(page, gfpflags))) + goto deactivate_slab; - /* must check again c->freelist in case of cpu migration or IRQ */ + /* must check again c->page in case IRQ handler changed it */ + local_irq_save(flags); + if (unlikely(page != c->page)) { + local_irq_restore(flags); + goto reread_page; + } freelist = c->freelist; if (freelist) goto load_freelist; @@ -2677,6 +2685,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, stat(s, ALLOC_REFILL); load_freelist: + + lockdep_assert_irqs_disabled(); + /* * freelist is pointing to the list of objects to be used. * page is pointing to the page from which the objects are obtained. @@ -2688,11 +2699,23 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, local_irq_restore(flags); return freelist; +deactivate_slab: + + local_irq_save(flags); + if (page != c->page) { + local_irq_restore(flags); + goto reread_page; + } + deactivate_slab(s, page, c->freelist, c); + new_slab: + lockdep_assert_irqs_disabled(); + if (slub_percpu_partial(c)) { page = c->page = slub_percpu_partial(c); slub_set_percpu_partial(c, page); + local_irq_restore(flags); stat(s, CPU_PARTIAL_ALLOC); goto redo; } From patchwork Wed Jun 9 11:38:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1504C48BD1 for ; Wed, 9 Jun 2021 11:39:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 789C360FD8 for ; Wed, 9 Jun 2021 11:39:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 789C360FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CD23A6B0087; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BBE636B0088; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B2CC6B0088; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0243.hostedemail.com [216.40.44.243]) by kanga.kvack.org (Postfix) with ESMTP id 0534B6B007B for ; Wed, 9 Jun 2021 07:39:35 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 9CA37181AEF2A for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) X-FDA: 78233990310.10.0451993 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf23.hostedemail.com (Postfix) with ESMTP id D53E0A000247 for ; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 5409A1FD68; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ufCLnw446ARgJJPk9KebxnNiU/kr4b5YrNsPcsD6jYQ=; b=ZCNw0yccEZ8Xvn7x28T67xwao91S9+d40P9Zmc9Q0graNP0VzIly8pjulg5dqbpS3OkLXa GRk7UtYiQqX1UqrEVDLhFXwsuU/fu8lrMk1xH3yC1HRxO+uQxotdGTvMYd2PUa/ubH2ROo 2tpqz4/bjPI55ZT+arcT5Gg6rz7NXuo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ufCLnw446ARgJJPk9KebxnNiU/kr4b5YrNsPcsD6jYQ=; b=uUS8PbDZQ3vjf+0vBxFfUtrq/RkVGkXpJ/yNGK6v3cDJ8ufqr1h55Ok9ZUMvPqC/4n+Ff6 3BctD7jrmKsYpVCg== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 24BE1118DD; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ufCLnw446ARgJJPk9KebxnNiU/kr4b5YrNsPcsD6jYQ=; b=ZCNw0yccEZ8Xvn7x28T67xwao91S9+d40P9Zmc9Q0graNP0VzIly8pjulg5dqbpS3OkLXa GRk7UtYiQqX1UqrEVDLhFXwsuU/fu8lrMk1xH3yC1HRxO+uQxotdGTvMYd2PUa/ubH2ROo 2tpqz4/bjPI55ZT+arcT5Gg6rz7NXuo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ufCLnw446ARgJJPk9KebxnNiU/kr4b5YrNsPcsD6jYQ=; b=uUS8PbDZQ3vjf+0vBxFfUtrq/RkVGkXpJ/yNGK6v3cDJ8ufqr1h55Ok9ZUMvPqC/4n+Ff6 3BctD7jrmKsYpVCg== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id WIR6CHaowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:34 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 14/34] mm, slub: move disabling irqs closer to get_partial() in ___slab_alloc() Date: Wed, 9 Jun 2021 13:38:43 +0200 Message-Id: <20210609113903.1421-15-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ZCNw0ycc; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uUS8PbDZ; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ZCNw0ycc; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=uUS8PbDZ; dmarc=none; spf=pass (imf23.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Stat-Signature: 19m9ps6gr74ckrui9sdwfawu33ybwhn8 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D53E0A000247 X-HE-Tag: 1623238771-815853 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Continue reducing the irq disabled scope. Check for per-cpu partial slabs with first with irqs enabled and then recheck with irqs disabled before grabbing the slab page. Mostly preparatory for the following patches. Signed-off-by: Vlastimil Babka --- mm/slub.c | 34 +++++++++++++++++++++++++--------- 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 22b5bfef8aa6..acb5e8f1d9da 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2633,11 +2633,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (unlikely(node != NUMA_NO_NODE && !node_isset(node, slab_nodes))) node = NUMA_NO_NODE; - local_irq_save(flags); - if (unlikely(c->page)) { - local_irq_restore(flags); - goto reread_page; - } goto new_slab; } redo: @@ -2678,6 +2673,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (!freelist) { c->page = NULL; + local_irq_restore(flags); stat(s, DEACTIVATE_BYPASS); goto new_slab; } @@ -2707,12 +2703,19 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto reread_page; } deactivate_slab(s, page, c->freelist, c); + local_irq_restore(flags); new_slab: - lockdep_assert_irqs_disabled(); - if (slub_percpu_partial(c)) { + local_irq_save(flags); + if (unlikely(c->page)) { + local_irq_restore(flags); + goto reread_page; + } + if (unlikely(!slub_percpu_partial(c))) + goto new_objects; /* stolen by an IRQ handler */ + page = c->page = slub_percpu_partial(c); slub_set_percpu_partial(c, page); local_irq_restore(flags); @@ -2720,6 +2723,16 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto redo; } + local_irq_save(flags); + if (unlikely(c->page)) { + local_irq_restore(flags); + goto reread_page; + } + +new_objects: + + lockdep_assert_irqs_disabled(); + freelist = get_partial(s, gfpflags, node, &page); if (freelist) { c->page = page; @@ -2752,15 +2765,18 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, check_new_page: if (kmem_cache_debug(s)) { - if (!alloc_debug_processing(s, page, freelist, addr)) + if (!alloc_debug_processing(s, page, freelist, addr)) { /* Slab failed checks. Next slab needed */ + c->page = NULL; + local_irq_restore(flags); goto new_slab; - else + } else { /* * For debug case, we don't load freelist so that all * allocations go through alloc_debug_processing() */ goto return_single; + } } if (unlikely(!pfmemalloc_match(page, gfpflags))) From patchwork Wed Jun 9 11:38:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D935C48BCD for ; Wed, 9 Jun 2021 11:40:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 43EAC61364 for ; Wed, 9 Jun 2021 11:40:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 43EAC61364 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 49A156B0082; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1CF286B0088; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A832D6B007E; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0058.hostedemail.com [216.40.44.58]) by kanga.kvack.org (Postfix) with ESMTP id 4E3126B0082 for ; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id EB4C9B9E8 for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) X-FDA: 78233990310.30.C8979E5 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf03.hostedemail.com (Postfix) with ESMTP id 4313EC00CBEE for ; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 84B1D1FD69; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tbfCejBm4aJ4eyiGJzBDybVMR3HougtHgUCP1fs72bo=; b=abQ+3zq7aibBT/g90YxFKE7rlf+NZ9S9FZTxwKAh2QuzmHAYSsa8uB5EK/eiHpxrHVrPxq pgjW2LElsW0P+47Y+kCK7ikqyJsPBOZvoJKHKcxqiVqsGrH2Qoat0SQPh+TJAnsFRSyH+B AjgejlpHfvZzB4AoT7DbzQM4CynrS/I= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tbfCejBm4aJ4eyiGJzBDybVMR3HougtHgUCP1fs72bo=; b=bqh+vEXLGfW4VC1I3METAhg3INHSZVUllLrlgqb3HKqRkpcukg/2P3UhNoA4Ky7WoOrqok A7E8UUQ37dmjHKBA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 55DBC11A98; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tbfCejBm4aJ4eyiGJzBDybVMR3HougtHgUCP1fs72bo=; b=abQ+3zq7aibBT/g90YxFKE7rlf+NZ9S9FZTxwKAh2QuzmHAYSsa8uB5EK/eiHpxrHVrPxq pgjW2LElsW0P+47Y+kCK7ikqyJsPBOZvoJKHKcxqiVqsGrH2Qoat0SQPh+TJAnsFRSyH+B AjgejlpHfvZzB4AoT7DbzQM4CynrS/I= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tbfCejBm4aJ4eyiGJzBDybVMR3HougtHgUCP1fs72bo=; b=bqh+vEXLGfW4VC1I3METAhg3INHSZVUllLrlgqb3HKqRkpcukg/2P3UhNoA4Ky7WoOrqok A7E8UUQ37dmjHKBA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id QLR+FHaowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:34 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 15/34] mm, slub: restore irqs around calling new_slab() Date: Wed, 9 Jun 2021 13:38:44 +0200 Message-Id: <20210609113903.1421-16-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=abQ+3zq7; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=bqh+vEXL; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=abQ+3zq7; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=bqh+vEXL; dmarc=none; spf=pass (imf03.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: mgndtiiua68is1tp9wj4g9ukk9y17eh7 X-Rspamd-Queue-Id: 4313EC00CBEE X-HE-Tag: 1623238771-649578 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: allocate_slab() currently re-enables irqs before calling to the page allocator. It depends on gfpflags_allow_blocking() to determine if it's safe to do so. Now we can instead simply restore irq before calling it through new_slab(). The other caller early_kmem_cache_node_alloc() is unaffected by this. Signed-off-by: Vlastimil Babka --- mm/slub.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index acb5e8f1d9da..f4ce372e3dd3 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1749,9 +1749,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) flags &= gfp_allowed_mask; - if (gfpflags_allow_blocking(flags)) - local_irq_enable(); - flags |= s->allocflags; /* @@ -1810,8 +1807,6 @@ static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) page->frozen = 1; out: - if (gfpflags_allow_blocking(flags)) - local_irq_disable(); if (!page) return NULL; @@ -2739,16 +2734,17 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto check_new_page; } + local_irq_restore(flags); put_cpu_ptr(s->cpu_slab); page = new_slab(s, gfpflags, node); c = get_cpu_ptr(s->cpu_slab); if (unlikely(!page)) { - local_irq_restore(flags); slab_out_of_memory(s, gfpflags, node); return NULL; } + local_irq_save(flags); if (c->page) flush_slab(s, c); From patchwork Wed Jun 9 11:38:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0996C48BDF for ; Wed, 9 Jun 2021 11:40:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 56CD060FD8 for ; Wed, 9 Jun 2021 11:40:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 56CD060FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 21E536B007E; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F3C526B0082; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0CED6B0083; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0195.hostedemail.com [216.40.44.195]) by kanga.kvack.org (Postfix) with ESMTP id 600706B0089 for ; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0F7F9824999B for ; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) X-FDA: 78233990352.05.D1E6752 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf13.hostedemail.com (Postfix) with ESMTP id EC18BE000253 for ; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B2A47219A0; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tMRaEZFMqbqCquk4QWerCUuNLFI/dmGIIEo12matNCQ=; b=JixlPVZ/dhqgk2DKFXTb9/92CPbymu2U2qdnX76atqNlB3g4f5zRYLrlagVZykKDgTLiih vdtQ2g8HU6gvUexrcHtoj++v92tsyscySr+X6yT768ECqYJ9UoxdH2tvJvzE81A8ZjkZJl DiBFYN5o+iGwciHTh9oG8bqhimuctos= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tMRaEZFMqbqCquk4QWerCUuNLFI/dmGIIEo12matNCQ=; b=zfxQHLpAV+0vuOAM4N2A1QLCJuhr4WM5p+V9/v0tkidBV+a2F3kGHCnPj1z3qv/ypM1tUg uCtQ1DyLEASbacBw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 85851118DD; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tMRaEZFMqbqCquk4QWerCUuNLFI/dmGIIEo12matNCQ=; b=JixlPVZ/dhqgk2DKFXTb9/92CPbymu2U2qdnX76atqNlB3g4f5zRYLrlagVZykKDgTLiih vdtQ2g8HU6gvUexrcHtoj++v92tsyscySr+X6yT768ECqYJ9UoxdH2tvJvzE81A8ZjkZJl DiBFYN5o+iGwciHTh9oG8bqhimuctos= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tMRaEZFMqbqCquk4QWerCUuNLFI/dmGIIEo12matNCQ=; b=zfxQHLpAV+0vuOAM4N2A1QLCJuhr4WM5p+V9/v0tkidBV+a2F3kGHCnPj1z3qv/ypM1tUg uCtQ1DyLEASbacBw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id gI0YIHaowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:34 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 16/34] mm, slub: validate slab from partial list or page allocator before making it cpu slab Date: Wed, 9 Jun 2021 13:38:45 +0200 Message-Id: <20210609113903.1421-17-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="JixlPVZ/"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=zfxQHLpA; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="JixlPVZ/"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=zfxQHLpA; spf=pass (imf13.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: EC18BE000253 X-Stat-Signature: mo4bg8787w3z1rwocw5jcwwuf4xghqpb X-HE-Tag: 1623238771-697610 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When we obtain a new slab page from node partial list or page allocator, we assign it to kmem_cache_cpu, perform some checks, and if they fail, we undo the assignment. In order to allow doing the checks without irq disabled, restructure the code so that the checks are done first, and kmem_cache_cpu.page assignment only after they pass. Signed-off-by: Vlastimil Babka --- mm/slub.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index f4ce372e3dd3..378f1cb040b3 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2729,10 +2729,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, lockdep_assert_irqs_disabled(); freelist = get_partial(s, gfpflags, node, &page); - if (freelist) { - c->page = page; + if (freelist) goto check_new_page; - } local_irq_restore(flags); put_cpu_ptr(s->cpu_slab); @@ -2745,9 +2743,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, } local_irq_save(flags); - if (c->page) - flush_slab(s, c); - /* * No other reference to the page yet so we can * muck around with it freely without cmpxchg @@ -2756,14 +2751,12 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, page->freelist = NULL; stat(s, ALLOC_SLAB); - c->page = page; check_new_page: if (kmem_cache_debug(s)) { if (!alloc_debug_processing(s, page, freelist, addr)) { /* Slab failed checks. Next slab needed */ - c->page = NULL; local_irq_restore(flags); goto new_slab; } else { @@ -2782,10 +2775,18 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, */ goto return_single; + if (unlikely(c->page)) + flush_slab(s, c); + c->page = page; + goto load_freelist; return_single: + if (unlikely(c->page)) + flush_slab(s, c); + c->page = page; + deactivate_slab(s, page, get_freepointer(s, freelist), c); local_irq_restore(flags); return freelist; From patchwork Wed Jun 9 11:38:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 489B6C48BCF for ; Wed, 9 Jun 2021 11:40:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F1EEC60FD8 for ; Wed, 9 Jun 2021 11:40:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F1EEC60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 759D26B0088; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 41DF56B008A; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA2186B0092; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0215.hostedemail.com [216.40.44.215]) by kanga.kvack.org (Postfix) with ESMTP id AF9406B0082 for ; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 475D9180AD81A for ; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) X-FDA: 78233990352.38.ADC5DF3 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf18.hostedemail.com (Postfix) with ESMTP id 7DC79200107A for ; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id E25B21FD6A; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=volf3/1v2fagmByxx16yvJir7+qLZbMW6i+3I7p6Sqc=; b=lr54bFG1gI4xnJeu9xxylzjCxMUOqNqFn/br4xXHq81J69rcBsj28VivKVSEAEArKZDAqs h4rhgbwAeCs851IARKYZ+p4ZCItRK5L6PHtYZxVglhNAxtFCxN+fYOeSoYthoCHh6aWBOj JWWQgmM/g94TpKtwJNw/LbxsKiSgcVc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=volf3/1v2fagmByxx16yvJir7+qLZbMW6i+3I7p6Sqc=; b=wA99Gg5Pd7WMagaeXluGX6o8cRtQqRLxmU2ZtrWMN2q8RXGYk1MIWLpRajaR3MkBkyPGrN cG2wR3V0vN3R36Cg== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id B391D11A98; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=volf3/1v2fagmByxx16yvJir7+qLZbMW6i+3I7p6Sqc=; b=lr54bFG1gI4xnJeu9xxylzjCxMUOqNqFn/br4xXHq81J69rcBsj28VivKVSEAEArKZDAqs h4rhgbwAeCs851IARKYZ+p4ZCItRK5L6PHtYZxVglhNAxtFCxN+fYOeSoYthoCHh6aWBOj JWWQgmM/g94TpKtwJNw/LbxsKiSgcVc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238774; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=volf3/1v2fagmByxx16yvJir7+qLZbMW6i+3I7p6Sqc=; b=wA99Gg5Pd7WMagaeXluGX6o8cRtQqRLxmU2ZtrWMN2q8RXGYk1MIWLpRajaR3MkBkyPGrN cG2wR3V0vN3R36Cg== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id ONReK3aowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:34 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 17/34] mm, slub: check new pages with restored irqs Date: Wed, 9 Jun 2021 13:38:46 +0200 Message-Id: <20210609113903.1421-18-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=lr54bFG1; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=wA99Gg5P; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=lr54bFG1; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=wA99Gg5P; spf=pass (imf18.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 7DC79200107A X-Stat-Signature: t7f4qas7ezspn13b3zmmaydh4ef6btn5 X-HE-Tag: 1623238772-927597 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Building on top of the previous patch, re-enable irqs before checking new pages. alloc_debug_processing() is now called with enabled irqs so we need to remove VM_BUG_ON(!irqs_disabled()); in check_slab() - there doesn't seem to be a need for it anyway. Signed-off-by: Vlastimil Babka --- mm/slub.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 378f1cb040b3..760080b58957 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -954,8 +954,6 @@ static int check_slab(struct kmem_cache *s, struct page *page) { int maxobj; - VM_BUG_ON(!irqs_disabled()); - if (!PageSlab(page)) { slab_err(s, page, "Not a valid slab page"); return 0; @@ -2729,10 +2727,10 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, lockdep_assert_irqs_disabled(); freelist = get_partial(s, gfpflags, node, &page); + local_irq_restore(flags); if (freelist) goto check_new_page; - local_irq_restore(flags); put_cpu_ptr(s->cpu_slab); page = new_slab(s, gfpflags, node); c = get_cpu_ptr(s->cpu_slab); @@ -2742,7 +2740,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, return NULL; } - local_irq_save(flags); /* * No other reference to the page yet so we can * muck around with it freely without cmpxchg @@ -2757,7 +2754,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (kmem_cache_debug(s)) { if (!alloc_debug_processing(s, page, freelist, addr)) { /* Slab failed checks. Next slab needed */ - local_irq_restore(flags); goto new_slab; } else { /* @@ -2775,6 +2771,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, */ goto return_single; + local_irq_save(flags); if (unlikely(c->page)) flush_slab(s, c); c->page = page; @@ -2783,6 +2780,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, return_single: + local_irq_save(flags); if (unlikely(c->page)) flush_slab(s, c); c->page = page; From patchwork Wed Jun 9 11:38:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E9E5C48BD1 for ; Wed, 9 Jun 2021 11:40:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B875260FD8 for ; Wed, 9 Jun 2021 11:40:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B875260FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 99C416B0089; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DA316B0081; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AA7B6B0089; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0055.hostedemail.com [216.40.44.55]) by kanga.kvack.org (Postfix) with ESMTP id C8D096B0081 for ; Wed, 9 Jun 2021 07:39:36 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6DF2581F3 for ; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) X-FDA: 78233990352.17.CC007BC Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf22.hostedemail.com (Postfix) with ESMTP id BD25FC005A20 for ; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 1DF76219E1; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z8v4ryVEKUVoC+Nx8Pww9inYHYBKUn8XDFXC2DN/iVs=; b=hQJ14+UFclogDrfchxtALGrb5AN/Qu62FILb5FBbctbbzOVCNPpeX3lA+btq04MsEKTbtO sG2m+jjPz+/USgzwFSNzGa9m0j4QhVSYXXO3sluq4oDNEJXlrAWSG5bsU9Hqa+X6X8Tgja WbMpLZ6NAa3hMrVvCY6VHG96k7uwcq8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z8v4ryVEKUVoC+Nx8Pww9inYHYBKUn8XDFXC2DN/iVs=; b=PaMxdXxvO6Exmk8d3FptcxglUOrdFSfEbkRN4Jnc8ppEBjJ3TDmap479lfL7G7XqP/mIJL 0Ri6gSjT6zL40BCg== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id E38DF118DD; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z8v4ryVEKUVoC+Nx8Pww9inYHYBKUn8XDFXC2DN/iVs=; b=hQJ14+UFclogDrfchxtALGrb5AN/Qu62FILb5FBbctbbzOVCNPpeX3lA+btq04MsEKTbtO sG2m+jjPz+/USgzwFSNzGa9m0j4QhVSYXXO3sluq4oDNEJXlrAWSG5bsU9Hqa+X6X8Tgja WbMpLZ6NAa3hMrVvCY6VHG96k7uwcq8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z8v4ryVEKUVoC+Nx8Pww9inYHYBKUn8XDFXC2DN/iVs=; b=PaMxdXxvO6Exmk8d3FptcxglUOrdFSfEbkRN4Jnc8ppEBjJ3TDmap479lfL7G7XqP/mIJL 0Ri6gSjT6zL40BCg== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id iHkcN3aowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:34 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 18/34] mm, slub: stop disabling irqs around get_partial() Date: Wed, 9 Jun 2021 13:38:47 +0200 Message-Id: <20210609113903.1421-19-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=hQJ14+UF; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=PaMxdXxv; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=hQJ14+UF; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=PaMxdXxv; dmarc=none; spf=pass (imf22.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: mj19d39ro77ntaqiw4zdt8runizxaxa8 X-Rspamd-Queue-Id: BD25FC005A20 X-HE-Tag: 1623238771-384826 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The function get_partial() does not need to have irqs disabled as a whole. It's sufficient to convert spin_lock operations to their irq saving/restoring versions. As a result, it's now possible to reach the page allocator from the slab allocator without disabling and re-enabling interrupts on the way. Signed-off-by: Vlastimil Babka --- mm/slub.c | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 760080b58957..6f894d1e84d6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1950,11 +1950,12 @@ static inline bool pfmemalloc_match(struct page *page, gfp_t gfpflags); * Try to allocate a partial slab from a specific node. */ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, - struct page **ret_page, gfp_t flags) + struct page **ret_page, gfp_t gfpflags) { struct page *page, *page2; void *object = NULL; unsigned int available = 0; + unsigned long flags; int objects; /* @@ -1966,11 +1967,11 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, if (!n || !n->nr_partial) return NULL; - spin_lock(&n->list_lock); + spin_lock_irqsave(&n->list_lock, flags); list_for_each_entry_safe(page, page2, &n->partial, slab_list) { void *t; - if (!pfmemalloc_match(page, flags)) + if (!pfmemalloc_match(page, gfpflags)) continue; t = acquire_slab(s, n, page, object == NULL, &objects); @@ -1991,7 +1992,7 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, break; } - spin_unlock(&n->list_lock); + spin_unlock_irqrestore(&n->list_lock, flags); return object; } @@ -2706,8 +2707,10 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, local_irq_restore(flags); goto reread_page; } - if (unlikely(!slub_percpu_partial(c))) + if (unlikely(!slub_percpu_partial(c))) { + local_irq_restore(flags); goto new_objects; /* stolen by an IRQ handler */ + } page = c->page = slub_percpu_partial(c); slub_set_percpu_partial(c, page); @@ -2716,18 +2719,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto redo; } - local_irq_save(flags); - if (unlikely(c->page)) { - local_irq_restore(flags); - goto reread_page; - } - new_objects: - lockdep_assert_irqs_disabled(); - freelist = get_partial(s, gfpflags, node, &page); - local_irq_restore(flags); if (freelist) goto check_new_page; From patchwork Wed Jun 9 11:38:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6193C48BDF for ; Wed, 9 Jun 2021 11:40:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 573FC60FD8 for ; Wed, 9 Jun 2021 11:40:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 573FC60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C5A416B0092; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE3AC6B0081; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9254B6B008C; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0098.hostedemail.com [216.40.44.98]) by kanga.kvack.org (Postfix) with ESMTP id 1ABF66B0093 for ; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: from smtpin40.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B03EF181AEF2A for ; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) X-FDA: 78233990352.40.8880054 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf17.hostedemail.com (Postfix) with ESMTP id 7FA024202A0A for ; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 4C0CC1FD6B; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cqt0VXEchackCh9ETD6KPbwS56wP8+xgoNpRJiEk95Q=; b=gpYYQAKtd2Lwif+d4UZBHLgSL8vuuV+4zM4kQ1Ln6B6RZo4EnFEmpmcKrw2lXGLdfKpb+c SMseFmoRl3zVfWtFjvyKxgofvLyau2t+voqxPqCHPcoq+ZnWfH2T2fq6eGqW393QZmaov2 1zwd7rQt7dwHJtDDfd34O/+c0FrWpXo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cqt0VXEchackCh9ETD6KPbwS56wP8+xgoNpRJiEk95Q=; b=okCO1DzMKaLPjSNK2ZYSOvomn80slibDBV2mV+Y/ePmltdzlc1O1KSDFI5Ln4UV+YEhYzz EgVrDFJlomag1GBw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 1E02811A98; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cqt0VXEchackCh9ETD6KPbwS56wP8+xgoNpRJiEk95Q=; b=gpYYQAKtd2Lwif+d4UZBHLgSL8vuuV+4zM4kQ1Ln6B6RZo4EnFEmpmcKrw2lXGLdfKpb+c SMseFmoRl3zVfWtFjvyKxgofvLyau2t+voqxPqCHPcoq+ZnWfH2T2fq6eGqW393QZmaov2 1zwd7rQt7dwHJtDDfd34O/+c0FrWpXo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Cqt0VXEchackCh9ETD6KPbwS56wP8+xgoNpRJiEk95Q=; b=okCO1DzMKaLPjSNK2ZYSOvomn80slibDBV2mV+Y/ePmltdzlc1O1KSDFI5Ln4UV+YEhYzz EgVrDFJlomag1GBw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id MMjdBneowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:35 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 19/34] mm, slub: move reset of c->page and freelist out of deactivate_slab() Date: Wed, 9 Jun 2021 13:38:48 +0200 Message-Id: <20210609113903.1421-20-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=gpYYQAKt; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=okCO1DzM; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=gpYYQAKt; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=okCO1DzM; dmarc=none; spf=pass (imf17.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Stat-Signature: tjm8bqjbpo5iuqnghkmzot5rab9o7kec X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7FA024202A0A X-HE-Tag: 1623238773-551763 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: deactivate_slab() removes the cpu slab by merging the cpu freelist with slab's freelist and putting the slab on the proper node's list. It also sets the respective kmem_cache_cpu pointers to NULL. By extracting the kmem_cache_cpu operations from the function, we can make it not dependent on disabled irqs. Also if we return a single free pointer from ___slab_alloc, we no longer have to assign kmem_cache_cpu.page before deactivation or care if somebody preempted us and assigned a different page to our kmem_cache_cpu in the process. Signed-off-by: Vlastimil Babka --- mm/slub.c | 31 ++++++++++++++++++------------- 1 file changed, 18 insertions(+), 13 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 6f894d1e84d6..8b0b975bfa36 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2149,10 +2149,13 @@ static void init_kmem_cache_cpus(struct kmem_cache *s) } /* - * Remove the cpu slab + * Finishes removing the cpu slab. Merges cpu's freelist with page's freelist, + * unfreezes the slabs and puts it on the proper list. + * Assumes the slab has been already safely taken away from kmem_cache_cpu + * by the caller. */ static void deactivate_slab(struct kmem_cache *s, struct page *page, - void *freelist, struct kmem_cache_cpu *c) + void *freelist) { enum slab_modes { M_NONE, M_PARTIAL, M_FULL, M_FREE }; struct kmem_cache_node *n = get_node(s, page_to_nid(page)); @@ -2281,9 +2284,6 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, discard_slab(s, page); stat(s, FREE_SLAB); } - - c->page = NULL; - c->freelist = NULL; } /* @@ -2408,10 +2408,16 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c) { - stat(s, CPUSLAB_FLUSH); - deactivate_slab(s, c->page, c->freelist, c); + void *freelist = c->freelist; + struct page *page = c->page; + c->page = NULL; + c->freelist = NULL; c->tid = next_tid(c->tid); + + deactivate_slab(s, page, freelist); + + stat(s, CPUSLAB_FLUSH); } /* @@ -2696,7 +2702,10 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, local_irq_restore(flags); goto reread_page; } - deactivate_slab(s, page, c->freelist, c); + freelist = c->freelist; + c->page = NULL; + c->freelist = NULL; + deactivate_slab(s, page, freelist); local_irq_restore(flags); new_slab: @@ -2775,11 +2784,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, return_single: local_irq_save(flags); - if (unlikely(c->page)) - flush_slab(s, c); - c->page = page; - - deactivate_slab(s, page, get_freepointer(s, freelist), c); + deactivate_slab(s, page, get_freepointer(s, freelist)); local_irq_restore(flags); return freelist; } From patchwork Wed Jun 9 11:38:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A510C48BCD for ; Wed, 9 Jun 2021 11:40:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4C4CC60FD8 for ; Wed, 9 Jun 2021 11:40:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4C4CC60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 04DE96B0095; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E519D6B008A; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1E246B0098; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0098.hostedemail.com [216.40.44.98]) by kanga.kvack.org (Postfix) with ESMTP id 62DB86B0092 for ; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D7D78BA10 for ; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) X-FDA: 78233990352.20.6DA9D97 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf11.hostedemail.com (Postfix) with ESMTP id 09F3C20010A9 for ; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 7B17E219E5; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=epQKKcaNSs6oc8v4JtInuf1QdvJlAwnRMC1xTjWgeXI=; b=WO97LCf9vSIMTWJg/0ZzucZdzc/KUz0BS36mpkEr6IkncBpRRtXErcnJjbNRWwqvcSoK4a YRPFd1XsAFa46zoD6I2jXpss1OADAEXhNgqpZn3dqego9LkLzkcAexp6GkyMgMZRpCkb4g jJWjhtBNnoO6kXeDbX2Ldl+z4XFJTls= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=epQKKcaNSs6oc8v4JtInuf1QdvJlAwnRMC1xTjWgeXI=; b=yg5PRqXC80SpIoAHlB81o2BuIzmk+uejuwQLDb2xD9TU/xKj7uCw3rpzFLHbFm67QcSHAq 93Be0I1JgmWfW8Aw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 4DCB3118DD; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=epQKKcaNSs6oc8v4JtInuf1QdvJlAwnRMC1xTjWgeXI=; b=WO97LCf9vSIMTWJg/0ZzucZdzc/KUz0BS36mpkEr6IkncBpRRtXErcnJjbNRWwqvcSoK4a YRPFd1XsAFa46zoD6I2jXpss1OADAEXhNgqpZn3dqego9LkLzkcAexp6GkyMgMZRpCkb4g jJWjhtBNnoO6kXeDbX2Ldl+z4XFJTls= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=epQKKcaNSs6oc8v4JtInuf1QdvJlAwnRMC1xTjWgeXI=; b=yg5PRqXC80SpIoAHlB81o2BuIzmk+uejuwQLDb2xD9TU/xKj7uCw3rpzFLHbFm67QcSHAq 93Be0I1JgmWfW8Aw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id YM1+EneowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:35 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 20/34] mm, slub: make locking in deactivate_slab() irq-safe Date: Wed, 9 Jun 2021 13:38:49 +0200 Message-Id: <20210609113903.1421-21-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=WO97LCf9; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yg5PRqXC; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=WO97LCf9; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=yg5PRqXC; spf=pass (imf11.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 09F3C20010A9 X-Stat-Signature: xmaya6uy6w6hey4fdad989fhhjgsfs3w X-HE-Tag: 1623238771-214948 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: dectivate_slab() now no longer touches the kmem_cache_cpu structure, so it will be possible to call it with irqs enabled. Just convert the spin_lock calls to their irq saving/restoring variants to make it irq-safe. Note we now have to use cmpxchg_double_slab() for irq-safe slab_lock(), because in some situations we don't take the list_lock, which would disable irqs. Signed-off-by: Vlastimil Babka --- mm/slub.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 8b0b975bfa36..bba18146def6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2163,6 +2163,7 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, enum slab_modes l = M_NONE, m = M_NONE; void *nextfree, *freelist_iter, *freelist_tail; int tail = DEACTIVATE_TO_HEAD; + unsigned long flags = 0; struct page new; struct page old; @@ -2238,7 +2239,7 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, * that acquire_slab() will see a slab page that * is frozen */ - spin_lock(&n->list_lock); + spin_lock_irqsave(&n->list_lock, flags); } } else { m = M_FULL; @@ -2249,7 +2250,7 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, * slabs from diagnostic functions will not see * any frozen slabs. */ - spin_lock(&n->list_lock); + spin_lock_irqsave(&n->list_lock, flags); } } @@ -2266,14 +2267,14 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, } l = m; - if (!__cmpxchg_double_slab(s, page, + if (!cmpxchg_double_slab(s, page, old.freelist, old.counters, new.freelist, new.counters, "unfreezing slab")) goto redo; if (lock) - spin_unlock(&n->list_lock); + spin_unlock_irqrestore(&n->list_lock, flags); if (m == M_PARTIAL) stat(s, tail); From patchwork Wed Jun 9 11:38:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309869 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E380C48BD1 for ; Wed, 9 Jun 2021 11:40:13 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 11F2360FD8 for ; Wed, 9 Jun 2021 11:40:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 11F2360FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3F1176B008A; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 10D8D6B0081; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C57546B008C; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0179.hostedemail.com [216.40.44.179]) by kanga.kvack.org (Postfix) with ESMTP id 64E2A6B0095 for ; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0EFFC181AEF30 for ; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) X-FDA: 78233990394.02.A50C337 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf05.hostedemail.com (Postfix) with ESMTP id 83A47E000265 for ; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id AB664219E6; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8TMqHwk78Ooa/tKgDwDLFTYj62DqSJ+/+iH5D+PQaTw=; b=xNml2b/xZFeXKSMw0bboB495jc9H0LIgaPrTmQ9wZF7BL6+w/h/ygalY3Wp7yJNfhg0Rp9 /G25JBQFDmIoLqMT1lYUlmtnvPoKOYrIOyA+3ecwHSqOI3OBYztII5j0U1u7b2F3R5fA/9 2gi40po9/AlLuEJT4DTgFLOsCnL3pps= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8TMqHwk78Ooa/tKgDwDLFTYj62DqSJ+/+iH5D+PQaTw=; b=aRLQO7bhseIQlT/OaIBIyWHymfr1355G1zpwuBu8to6DqC5r/TZte6vvrjL93BAKyBoCjs ZJcw9FQY0eSgBBAA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 7C72F11A98; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8TMqHwk78Ooa/tKgDwDLFTYj62DqSJ+/+iH5D+PQaTw=; b=xNml2b/xZFeXKSMw0bboB495jc9H0LIgaPrTmQ9wZF7BL6+w/h/ygalY3Wp7yJNfhg0Rp9 /G25JBQFDmIoLqMT1lYUlmtnvPoKOYrIOyA+3ecwHSqOI3OBYztII5j0U1u7b2F3R5fA/9 2gi40po9/AlLuEJT4DTgFLOsCnL3pps= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8TMqHwk78Ooa/tKgDwDLFTYj62DqSJ+/+iH5D+PQaTw=; b=aRLQO7bhseIQlT/OaIBIyWHymfr1355G1zpwuBu8to6DqC5r/TZte6vvrjL93BAKyBoCjs ZJcw9FQY0eSgBBAA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id kDrlHXeowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:35 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 21/34] mm, slub: call deactivate_slab() without disabling irqs Date: Wed, 9 Jun 2021 13:38:50 +0200 Message-Id: <20210609113903.1421-22-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="xNml2b/x"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=aRLQO7bh; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="xNml2b/x"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=aRLQO7bh; spf=pass (imf05.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: wyd3ew8ufp7daa4sred6mwhstt3fgory X-Rspamd-Queue-Id: 83A47E000265 X-Rspamd-Server: rspam06 X-HE-Tag: 1623238771-260767 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The function is now safe to be called with irqs enabled, so move the calls outside of irq disabled sections. When called from ___slab_alloc() -> flush_slab() we have irqs disabled, so to reenable them before deactivate_slab() we need to open-code flush_slab() in ___slab_alloc() and reenable irqs after modifying the kmem_cache_cpu fields. But that means a IRQ handler meanwhile might have assigned a new page to kmem_cache_cpu.page so we have to retry the whole check. The remaining callers of flush_slab() are the IPI handler which has disabled irqs anyway, and slub_cpu_dead() which will be dealt with in the following patch. Signed-off-by: Vlastimil Babka --- mm/slub.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index bba18146def6..a99a254f2410 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2706,8 +2706,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, freelist = c->freelist; c->page = NULL; c->freelist = NULL; - deactivate_slab(s, page, freelist); local_irq_restore(flags); + deactivate_slab(s, page, freelist); new_slab: @@ -2775,18 +2775,32 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, */ goto return_single; +retry_load_page: + local_irq_save(flags); - if (unlikely(c->page)) - flush_slab(s, c); + if (unlikely(c->page)) { + void *flush_freelist = c->freelist; + struct page *flush_page = c->page; + + c->page = NULL; + c->freelist = NULL; + c->tid = next_tid(c->tid); + + local_irq_restore(flags); + + deactivate_slab(s, flush_page, flush_freelist); + + stat(s, CPUSLAB_FLUSH); + + goto retry_load_page; + } c->page = page; goto load_freelist; return_single: - local_irq_save(flags); deactivate_slab(s, page, get_freepointer(s, freelist)); - local_irq_restore(flags); return freelist; } From patchwork Wed Jun 9 11:38:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309871 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BB8AC48BCD for ; Wed, 9 Jun 2021 11:40:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C498E613B1 for ; Wed, 9 Jun 2021 11:40:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C498E613B1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 72AA16B0093; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 30B336B0098; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 049E46B0093; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0229.hostedemail.com [216.40.44.229]) by kanga.kvack.org (Postfix) with ESMTP id AE07B6B0096 for ; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 591DE824999B for ; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) X-FDA: 78233990394.14.277EC57 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf24.hostedemail.com (Postfix) with ESMTP id C643DA00025F for ; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id DE7951FD6C; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q4H+O6ZFlXECn/j2lYWRNSXW+SmHmqvp1ZhSwxPI3Jw=; b=ZC9wIZSPPZ24y4U1q5vUZfHTz84l/8N5U2Ax2XOLWhjYGtaJQLC8erCQhKhO+QauwuWH5w dln/ljCmlVZERrlh3Pw8i2+UfrYvQzzzsCObf2uojkNZzWWAjsbFlK39FEVTjQ34JO7mXf bu7c3GDxUmrgXbc1O8QT+I4ocoJH/IM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q4H+O6ZFlXECn/j2lYWRNSXW+SmHmqvp1ZhSwxPI3Jw=; b=Ts2r0o78WKFso4GH8IJlkQM6/kQ01FT+XM3VFmg/VZ9SnbTug3DsK+Zog+T4ooO85T2DMh /SS6Ce2bcXUHYqCQ== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id AD014118DD; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q4H+O6ZFlXECn/j2lYWRNSXW+SmHmqvp1ZhSwxPI3Jw=; b=ZC9wIZSPPZ24y4U1q5vUZfHTz84l/8N5U2Ax2XOLWhjYGtaJQLC8erCQhKhO+QauwuWH5w dln/ljCmlVZERrlh3Pw8i2+UfrYvQzzzsCObf2uojkNZzWWAjsbFlK39FEVTjQ34JO7mXf bu7c3GDxUmrgXbc1O8QT+I4ocoJH/IM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238775; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=q4H+O6ZFlXECn/j2lYWRNSXW+SmHmqvp1ZhSwxPI3Jw=; b=Ts2r0o78WKFso4GH8IJlkQM6/kQ01FT+XM3VFmg/VZ9SnbTug3DsK+Zog+T4ooO85T2DMh /SS6Ce2bcXUHYqCQ== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id yNy4KXeowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:35 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 22/34] mm, slub: move irq control into unfreeze_partials() Date: Wed, 9 Jun 2021 13:38:51 +0200 Message-Id: <20210609113903.1421-23-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ZC9wIZSP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Ts2r0o78; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ZC9wIZSP; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Ts2r0o78; dmarc=none; spf=pass (imf24.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Stat-Signature: 9gbenkz6uorobogzqy58qqjny6ope3fa X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: C643DA00025F X-HE-Tag: 1623238772-640034 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: unfreeze_partials() can be optimized so that it doesn't need irqs disabled for the whole time. As the first step, move irq control into the function and remove it from the put_cpu_partial() caller. Signed-off-by: Vlastimil Babka --- mm/slub.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index a99a254f2410..4a3cd8174cd2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2290,9 +2290,8 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, /* * Unfreeze all the cpu partial slabs. * - * This function must be called with interrupts disabled - * for the cpu using c (or some other guarantee must be there - * to guarantee no concurrent accesses). + * This function must be called with preemption or migration + * disabled with c local to the cpu. */ static void unfreeze_partials(struct kmem_cache *s, struct kmem_cache_cpu *c) @@ -2300,6 +2299,9 @@ static void unfreeze_partials(struct kmem_cache *s, #ifdef CONFIG_SLUB_CPU_PARTIAL struct kmem_cache_node *n = NULL, *n2 = NULL; struct page *page, *discard_page = NULL; + unsigned long flags; + + local_irq_save(flags); while ((page = slub_percpu_partial(c))) { struct page new; @@ -2352,6 +2354,8 @@ static void unfreeze_partials(struct kmem_cache *s, discard_slab(s, page); stat(s, FREE_SLAB); } + + local_irq_restore(flags); #endif /* CONFIG_SLUB_CPU_PARTIAL */ } @@ -2379,14 +2383,11 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) pobjects = oldpage->pobjects; pages = oldpage->pages; if (drain && pobjects > slub_cpu_partial(s)) { - unsigned long flags; /* * partial array is full. Move the existing * set to the per node partial list. */ - local_irq_save(flags); unfreeze_partials(s, this_cpu_ptr(s->cpu_slab)); - local_irq_restore(flags); oldpage = NULL; pobjects = 0; pages = 0; From patchwork Wed Jun 9 11:38:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8481C48BCD for ; Wed, 9 Jun 2021 11:40:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6B9C360FD8 for ; Wed, 9 Jun 2021 11:40:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6B9C360FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C818D6B0098; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B93496B009B; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 926BA6B0098; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0116.hostedemail.com [216.40.44.116]) by kanga.kvack.org (Postfix) with ESMTP id 2300F6B008C for ; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id BFC85B9E8 for ; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) X-FDA: 78233990394.11.71169DE Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf03.hostedemail.com (Postfix) with ESMTP id 020DDC00CBC0 for ; Wed, 9 Jun 2021 11:39:32 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 1AF66219E7; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CLObwQuENcRgoukMvNxRJzoRvFwUKPTdwZDaHjVrHHA=; b=YlGl6sq2LuhXUI56gbtKfae6pZ4AY5ckS09vFiOApOYSgzTheLO36HgdqBhxrpyvLXrv6H xouoMVF2jZrrd2Qj+k4B9VDu3nuRfGAPRnJE+XSP1Wozu8FXIR5jMNUUmTKpLqUlSt+jgv Z/yDLxcN+rOcHGR4MgtzhMN/GThiauA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CLObwQuENcRgoukMvNxRJzoRvFwUKPTdwZDaHjVrHHA=; b=Tf1wjSQaqv/yJ3IoWAa7MJQE8Qp0aY6MDq08lGLOT3kASNsCN2/NXpw1KdzDwsUIYOjxTa N68AarkIwtKZCbAw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id E01C611A98; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CLObwQuENcRgoukMvNxRJzoRvFwUKPTdwZDaHjVrHHA=; b=YlGl6sq2LuhXUI56gbtKfae6pZ4AY5ckS09vFiOApOYSgzTheLO36HgdqBhxrpyvLXrv6H xouoMVF2jZrrd2Qj+k4B9VDu3nuRfGAPRnJE+XSP1Wozu8FXIR5jMNUUmTKpLqUlSt+jgv Z/yDLxcN+rOcHGR4MgtzhMN/GThiauA= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CLObwQuENcRgoukMvNxRJzoRvFwUKPTdwZDaHjVrHHA=; b=Tf1wjSQaqv/yJ3IoWAa7MJQE8Qp0aY6MDq08lGLOT3kASNsCN2/NXpw1KdzDwsUIYOjxTa N68AarkIwtKZCbAw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id uFI8NneowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:35 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 23/34] mm, slub: discard slabs in unfreeze_partials() without irqs disabled Date: Wed, 9 Jun 2021 13:38:52 +0200 Message-Id: <20210609113903.1421-24-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=YlGl6sq2; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Tf1wjSQa; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=YlGl6sq2; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=Tf1wjSQa; dmarc=none; spf=pass (imf03.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: 5pmauy5n663j3479pmeefn19e95nifix X-Rspamd-Queue-Id: 020DDC00CBC0 X-HE-Tag: 1623238772-979001 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No need for disabled irqs when discarding slabs, so restore them before discarding. Signed-off-by: Vlastimil Babka --- mm/slub.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/slub.c b/mm/slub.c index 4a3cd8174cd2..dcbc5baea702 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2346,6 +2346,8 @@ static void unfreeze_partials(struct kmem_cache *s, if (n) spin_unlock(&n->list_lock); + local_irq_restore(flags); + while (discard_page) { page = discard_page; discard_page = discard_page->next; @@ -2355,7 +2357,6 @@ static void unfreeze_partials(struct kmem_cache *s, stat(s, FREE_SLAB); } - local_irq_restore(flags); #endif /* CONFIG_SLUB_CPU_PARTIAL */ } From patchwork Wed Jun 9 11:38:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDE56C48BD1 for ; Wed, 9 Jun 2021 11:40:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7556C60FD8 for ; Wed, 9 Jun 2021 11:40:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7556C60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9A93E6B008C; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C9586B0096; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A86B6B0099; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0096.hostedemail.com [216.40.44.96]) by kanga.kvack.org (Postfix) with ESMTP id 06E316B0096 for ; Wed, 9 Jun 2021 07:39:37 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 9F5A0181AEF2A for ; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) X-FDA: 78233990394.17.1AB8B66 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf23.hostedemail.com (Postfix) with ESMTP id C9960A00014E for ; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 4A74A1FD6D; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2GjVus2UBVIbubovFl8dtg2IvpbcICqCbHPjVLJTVoM=; b=o04cHXUdiGCGIfGonPs5+wcwkDX3KOGSlylgZ92FJt2EYgD7mTURBu8qyVVHAsNEAOWE4V q6drLCxA1yHq6/mHC8Wc7O3Mq5HVmLcYrSj81I7/VxeSDHhFvL8Mw6wOOmtTTDtbZOIL3t wRZmDqZHGINPKmx251+9SDrtFPMn+AI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2GjVus2UBVIbubovFl8dtg2IvpbcICqCbHPjVLJTVoM=; b=dfQb9bfoXcUTIavFmii6oJq++Hz+Aiv806ScHsu2USbjV7fDeKJt559qCUdaPGWsXHtY8e gz/V/lU1wR5fqpBg== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 1BE6D118DD; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2GjVus2UBVIbubovFl8dtg2IvpbcICqCbHPjVLJTVoM=; b=o04cHXUdiGCGIfGonPs5+wcwkDX3KOGSlylgZ92FJt2EYgD7mTURBu8qyVVHAsNEAOWE4V q6drLCxA1yHq6/mHC8Wc7O3Mq5HVmLcYrSj81I7/VxeSDHhFvL8Mw6wOOmtTTDtbZOIL3t wRZmDqZHGINPKmx251+9SDrtFPMn+AI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2GjVus2UBVIbubovFl8dtg2IvpbcICqCbHPjVLJTVoM=; b=dfQb9bfoXcUTIavFmii6oJq++Hz+Aiv806ScHsu2USbjV7fDeKJt559qCUdaPGWsXHtY8e gz/V/lU1wR5fqpBg== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id KDVWBniowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:36 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 24/34] mm, slub: detach whole partial list at once in unfreeze_partials() Date: Wed, 9 Jun 2021 13:38:53 +0200 Message-Id: <20210609113903.1421-25-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=o04cHXUd; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=dfQb9bfo; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=o04cHXUd; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=dfQb9bfo; spf=pass (imf23.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: 3ncakkoqid3k1neqd6ct6qp4mfuzcuuq X-Rspamd-Queue-Id: C9960A00014E X-Rspamd-Server: rspam06 X-HE-Tag: 1623238773-878304 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of iterating through the live percpu partial list, detach it from the kmem_cache_cpu at once. This is simpler and will allow further optimization. Signed-off-by: Vlastimil Babka --- mm/slub.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index dcbc5baea702..3e0ec7211d55 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2298,16 +2298,20 @@ static void unfreeze_partials(struct kmem_cache *s, { #ifdef CONFIG_SLUB_CPU_PARTIAL struct kmem_cache_node *n = NULL, *n2 = NULL; - struct page *page, *discard_page = NULL; + struct page *page, *partial_page, *discard_page = NULL; unsigned long flags; local_irq_save(flags); - while ((page = slub_percpu_partial(c))) { + partial_page = slub_percpu_partial(c); + c->partial = NULL; + + while (partial_page) { struct page new; struct page old; - slub_set_percpu_partial(c, page); + page = partial_page; + partial_page = page->next; n2 = get_node(s, page_to_nid(page)); if (n != n2) { From patchwork Wed Jun 9 11:38:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DD3CC48BCF for ; Wed, 9 Jun 2021 11:40:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 423FB613B1 for ; Wed, 9 Jun 2021 11:40:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 423FB613B1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F18ED6B009B; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E03206B009E; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B6C876B009D; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0072.hostedemail.com [216.40.44.72]) by kanga.kvack.org (Postfix) with ESMTP id 684316B0081 for ; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0BBE2180AD81A for ; Wed, 9 Jun 2021 11:39:38 +0000 (UTC) X-FDA: 78233990436.02.54C2BF2 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf21.hostedemail.com (Postfix) with ESMTP id 16899E000240 for ; Wed, 9 Jun 2021 11:39:31 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 817DC1FD6E; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KdJtA4YoE6ORCzqq1AaRT5tX40ysKLrzxToE3yWrpNc=; b=eBbXi4CkX7EMTUHOJqT8xqoB7XkTXBITgC8amj4Ft5A/zuTI+epj2MA2uNs/7r7BWwbr8Z SlS9+EJ0riQXBr0gUf6QENf1TU0lv2w/UQc1fihTkGXAlQWqtpLPz9kUTGZeS96IQNstRK TjPilB4ucJlLxv+7y6oDtv+Y4skJtXM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KdJtA4YoE6ORCzqq1AaRT5tX40ysKLrzxToE3yWrpNc=; b=9Bh+xb3bwIRXHzIn2wniiILwcrIoB8irmjGvkKTb7yikdN5thh/m6p+drBo7EMwbtJWjHG p5UrXNRUSFAvq8Bw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 4D20311A98; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KdJtA4YoE6ORCzqq1AaRT5tX40ysKLrzxToE3yWrpNc=; b=eBbXi4CkX7EMTUHOJqT8xqoB7XkTXBITgC8amj4Ft5A/zuTI+epj2MA2uNs/7r7BWwbr8Z SlS9+EJ0riQXBr0gUf6QENf1TU0lv2w/UQc1fihTkGXAlQWqtpLPz9kUTGZeS96IQNstRK TjPilB4ucJlLxv+7y6oDtv+Y4skJtXM= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KdJtA4YoE6ORCzqq1AaRT5tX40ysKLrzxToE3yWrpNc=; b=9Bh+xb3bwIRXHzIn2wniiILwcrIoB8irmjGvkKTb7yikdN5thh/m6p+drBo7EMwbtJWjHG p5UrXNRUSFAvq8Bw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id 6EFaEniowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:36 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 25/34] mm, slub: detach percpu partial list in unfreeze_partials() using this_cpu_cmpxchg() Date: Wed, 9 Jun 2021 13:38:54 +0200 Message-Id: <20210609113903.1421-26-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 16899E000240 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=eBbXi4Ck; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=9Bh+xb3b; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=eBbXi4Ck; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=9Bh+xb3b; spf=pass (imf21.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: b3cmfom6tdigy18xbtwu5kpu86xm4jjs X-HE-Tag: 1623238771-473627 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of relying on disabled irqs for atomicity when detaching the percpu partial list, we can use this_cpu_cmpxchg() and detach without irqs disabled. However, unfreeze_partials() can be also called from another cpu on behalf of a cpu that is being offlined, so we need to restructure the code accordingly: - __unfreeze_partials() is the bulk of unfreeze_partials() that processes the detached percpu partial list - unfreeze_partials() uses this_cpu_cmpxchg() to detach list from current cpu - unfreeze_partials_cpu() is to be called for the offlined cpu so it needs no protection, and is called from __flush_cpu_slab() - flush_cpu_slab() is for the local cpu thus it needs to call unfreeze_partials(). So it can't simply call __flush_cpu_slab(smp_processor_id()) anymore and we have to open-code it Signed-off-by: Vlastimil Babka --- mm/slub.c | 77 +++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 55 insertions(+), 22 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 3e0ec7211d55..0867801e3086 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2287,25 +2287,15 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, } } -/* - * Unfreeze all the cpu partial slabs. - * - * This function must be called with preemption or migration - * disabled with c local to the cpu. - */ -static void unfreeze_partials(struct kmem_cache *s, - struct kmem_cache_cpu *c) -{ #ifdef CONFIG_SLUB_CPU_PARTIAL +static void __unfreeze_partials(struct kmem_cache *s, struct page *partial_page) +{ struct kmem_cache_node *n = NULL, *n2 = NULL; - struct page *page, *partial_page, *discard_page = NULL; + struct page *page, *discard_page = NULL; unsigned long flags; local_irq_save(flags); - partial_page = slub_percpu_partial(c); - c->partial = NULL; - while (partial_page) { struct page new; struct page old; @@ -2360,10 +2350,49 @@ static void unfreeze_partials(struct kmem_cache *s, discard_slab(s, page); stat(s, FREE_SLAB); } +} -#endif /* CONFIG_SLUB_CPU_PARTIAL */ +/* + * Unfreeze all the cpu partial slabs. + * + * This function must be called with preemption or migration + * disabled. + */ +static void unfreeze_partials(struct kmem_cache *s) +{ + struct page *partial_page; + + do { + partial_page = this_cpu_read(s->cpu_slab->partial); + + } while (partial_page && + this_cpu_cmpxchg(s->cpu_slab->partial, partial_page, NULL) + != partial_page); + + if (partial_page) + __unfreeze_partials(s, partial_page); } +static void unfreeze_partials_cpu(struct kmem_cache *s, + struct kmem_cache_cpu *c) +{ + struct page *partial_page; + + partial_page = slub_percpu_partial(c); + c->partial = NULL; + + if (partial_page) + __unfreeze_partials(s, partial_page); +} + +#else /* CONFIG_SLUB_CPU_PARTIAL */ + +static void unfreeze_partials(struct kmem_cache *s) { } +static void unfreeze_partials_cpu(struct kmem_cache *s, + struct kmem_cache_cpu *c) { } + +#endif /* CONFIG_SLUB_CPU_PARTIAL */ + /* * Put a page that was just frozen (in __slab_free|get_partial_node) into a * partial page slot if available. @@ -2392,7 +2421,7 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) * partial array is full. Move the existing * set to the per node partial list. */ - unfreeze_partials(s, this_cpu_ptr(s->cpu_slab)); + unfreeze_partials(s); oldpage = NULL; pobjects = 0; pages = 0; @@ -2427,11 +2456,6 @@ static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c) stat(s, CPUSLAB_FLUSH); } -/* - * Flush cpu slab. - * - * Called from IPI handler with interrupts disabled. - */ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu); @@ -2439,14 +2463,23 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) if (c->page) flush_slab(s, c); - unfreeze_partials(s, c); + unfreeze_partials_cpu(s, c); } +/* + * Flush cpu slab. + * + * Called from IPI handler with interrupts disabled. + */ static void flush_cpu_slab(void *d) { struct kmem_cache *s = d; + struct kmem_cache_cpu *c = this_cpu_ptr(s->cpu_slab); - __flush_cpu_slab(s, smp_processor_id()); + if (c->page) + flush_slab(s, c); + + unfreeze_partials(s); } static bool has_cpu_slab(int cpu, void *info) From patchwork Wed Jun 9 11:38:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAD94C48BCD for ; Wed, 9 Jun 2021 11:40:22 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 56CDD60FD8 for ; Wed, 9 Jun 2021 11:40:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 56CDD60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1F7A76B0096; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F16A16B009A; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C7DF16B0096; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0149.hostedemail.com [216.40.44.149]) by kanga.kvack.org (Postfix) with ESMTP id 8DE796B009A for ; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 22E7B6D7C for ; Wed, 9 Jun 2021 11:39:38 +0000 (UTC) X-FDA: 78233990436.21.FC0317D Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf10.hostedemail.com (Postfix) with ESMTP id 3DFF041AD7B6 for ; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id B42731FD6F; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zyVMd6JuJll2CSwZVtypUoJOKYt0RuiadUow3LJITGI=; b=K9O/gGfopsAbKUt5QoB/ur6B/XWNA+WamO+Tjd+gPPbho3pCYARNDp9aMd4AkxtiKpi5Ho CoP6+KqI4DpOC6prRlGDaOx+ewgETUeIN4lv3ZYe2ZIqpPY2egb93A7YJ8lCnw0e2zyEtZ Xgq9+5FFac3bkwvlt58AX11q59y1ni0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zyVMd6JuJll2CSwZVtypUoJOKYt0RuiadUow3LJITGI=; b=sHNNiLfzjvpwDljIQ60meButSP0QmB0R/YE2ciWU7SwTf853LSAJMxDxQXgzVDIcjGgsML 8xgYX5GqnfTakBCA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 82B51118DD; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zyVMd6JuJll2CSwZVtypUoJOKYt0RuiadUow3LJITGI=; b=K9O/gGfopsAbKUt5QoB/ur6B/XWNA+WamO+Tjd+gPPbho3pCYARNDp9aMd4AkxtiKpi5Ho CoP6+KqI4DpOC6prRlGDaOx+ewgETUeIN4lv3ZYe2ZIqpPY2egb93A7YJ8lCnw0e2zyEtZ Xgq9+5FFac3bkwvlt58AX11q59y1ni0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zyVMd6JuJll2CSwZVtypUoJOKYt0RuiadUow3LJITGI=; b=sHNNiLfzjvpwDljIQ60meButSP0QmB0R/YE2ciWU7SwTf853LSAJMxDxQXgzVDIcjGgsML 8xgYX5GqnfTakBCA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id gI9xH3iowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:36 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 26/34] mm, slub: only disable irq with spin_lock in __unfreeze_partials() Date: Wed, 9 Jun 2021 13:38:55 +0200 Message-Id: <20210609113903.1421-27-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="K9O/gGfo"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=sHNNiLfz; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b="K9O/gGfo"; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=sHNNiLfz; dmarc=none; spf=pass (imf10.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Stat-Signature: yiqhqr85tsfdkxpa7yw84dn7pxx311n8 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3DFF041AD7B6 X-HE-Tag: 1623238774-451943 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __unfreeze_partials() no longer needs to have irqs disabled, except for making the spin_lock operations irq-safe, so convert the spin_locks operations and remove the separate irq handling. Signed-off-by: Vlastimil Babka --- mm/slub.c | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 0867801e3086..5a81189608e6 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2292,9 +2292,7 @@ static void __unfreeze_partials(struct kmem_cache *s, struct page *partial_page) { struct kmem_cache_node *n = NULL, *n2 = NULL; struct page *page, *discard_page = NULL; - unsigned long flags; - - local_irq_save(flags); + unsigned long flags = 0; while (partial_page) { struct page new; @@ -2306,10 +2304,10 @@ static void __unfreeze_partials(struct kmem_cache *s, struct page *partial_page) n2 = get_node(s, page_to_nid(page)); if (n != n2) { if (n) - spin_unlock(&n->list_lock); + spin_unlock_irqrestore(&n->list_lock, flags); n = n2; - spin_lock(&n->list_lock); + spin_lock_irqsave(&n->list_lock, flags); } do { @@ -2338,9 +2336,7 @@ static void __unfreeze_partials(struct kmem_cache *s, struct page *partial_page) } if (n) - spin_unlock(&n->list_lock); - - local_irq_restore(flags); + spin_unlock_irqrestore(&n->list_lock, flags); while (discard_page) { page = discard_page; From patchwork Wed Jun 9 11:38:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68075C48BD1 for ; Wed, 9 Jun 2021 11:40:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 20788613B1 for ; Wed, 9 Jun 2021 11:40:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 20788613B1 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3F9B26B0081; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 15C906B009C; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E788D6B0099; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0154.hostedemail.com [216.40.44.154]) by kanga.kvack.org (Postfix) with ESMTP id A15CF6B0099 for ; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4474D81F3 for ; Wed, 9 Jun 2021 11:39:38 +0000 (UTC) X-FDA: 78233990436.03.4383B1C Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf16.hostedemail.com (Postfix) with ESMTP id 4B16180192FE for ; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id E5990219EA; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YBcrguTXnI9hsctZTgQ7OZG0uV6yh0oH29eLHd7YhOM=; b=S9aDCy8NqeSbElUMJ7WzqpduEvWVpEeZGxboXI4yXCxlIEu7fG1uKF0z9lJHnm7rpwddij n+bVZPPSvlT349WpwhqNFVtX8b//dGxw4IcvqKw8avsziFGZSHgZ6f2LHEqpK7YFhw79LZ Ybbq1xJcy2WHL07mz5xEhmKZbLmb+dc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YBcrguTXnI9hsctZTgQ7OZG0uV6yh0oH29eLHd7YhOM=; b=rL6U8d/br+ltLXTLyZtABuUoy7FOyMm0b6uCrM6FmDpyc330LYQnO+vOXlpMJMsavg/6bQ 1wJyokoIUQFb+EDQ== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id B64C611A98; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YBcrguTXnI9hsctZTgQ7OZG0uV6yh0oH29eLHd7YhOM=; b=S9aDCy8NqeSbElUMJ7WzqpduEvWVpEeZGxboXI4yXCxlIEu7fG1uKF0z9lJHnm7rpwddij n+bVZPPSvlT349WpwhqNFVtX8b//dGxw4IcvqKw8avsziFGZSHgZ6f2LHEqpK7YFhw79LZ Ybbq1xJcy2WHL07mz5xEhmKZbLmb+dc= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238776; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YBcrguTXnI9hsctZTgQ7OZG0uV6yh0oH29eLHd7YhOM=; b=rL6U8d/br+ltLXTLyZtABuUoy7FOyMm0b6uCrM6FmDpyc330LYQnO+vOXlpMJMsavg/6bQ 1wJyokoIUQFb+EDQ== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id MKT/K3iowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:36 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 27/34] mm, slub: don't disable irqs in slub_cpu_dead() Date: Wed, 9 Jun 2021 13:38:56 +0200 Message-Id: <20210609113903.1421-28-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 4B16180192FE X-Stat-Signature: jffotx9qsepffu4gw1b56xebx3chy5my Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=S9aDCy8N; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="rL6U8d/b"; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=S9aDCy8N; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b="rL6U8d/b"; dmarc=none; spf=pass (imf16.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-HE-Tag: 1623238774-795726 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: slub_cpu_dead() cleans up for an offlined cpu from another cpu and calls only functions that are now irq safe, so we don't need to disable irqs anymore. Signed-off-by: Vlastimil Babka --- mm/slub.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 5a81189608e6..12d35d6a6eaa 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2498,14 +2498,10 @@ static void flush_all(struct kmem_cache *s) static int slub_cpu_dead(unsigned int cpu) { struct kmem_cache *s; - unsigned long flags; mutex_lock(&slab_mutex); - list_for_each_entry(s, &slab_caches, list) { - local_irq_save(flags); + list_for_each_entry(s, &slab_caches, list) __flush_cpu_slab(s, cpu); - local_irq_restore(flags); - } mutex_unlock(&slab_mutex); return 0; } From patchwork Wed Jun 9 11:38:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FC23C48BCF for ; Wed, 9 Jun 2021 11:40:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 03E5A60FD8 for ; Wed, 9 Jun 2021 11:40:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 03E5A60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 753C96B009A; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5CBB16B009E; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 382BE6B009D; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0032.hostedemail.com [216.40.44.32]) by kanga.kvack.org (Postfix) with ESMTP id D1BE46B0081 for ; Wed, 9 Jun 2021 07:39:38 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 6CE70824999B for ; Wed, 9 Jun 2021 11:39:38 +0000 (UTC) X-FDA: 78233990436.25.E6CDC85 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf22.hostedemail.com (Postfix) with ESMTP id C080CC00CBD7 for ; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 269E3219EB; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i+kXFJUxiAt8nF/gFOeGLQ15QjX96rJgGO1xp0O5Hkw=; b=EZ1sKebsm1bY5kIaC7UF9DQ1G7UTyLUSr9XMyF5zYN+EANnQpxhQm1tfdWkXRzv30znYUQ e2SXselrS/jgk97AcPczw3+LhCZJhHsIaF93qGh7JBJZJviOst6a3gCaRPF+ca4dF6YusJ Uuo9uoa4l0+l44z3JCV12cZIzfG1Rbk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i+kXFJUxiAt8nF/gFOeGLQ15QjX96rJgGO1xp0O5Hkw=; b=43aWP7ErEcQ8m4WYdN2jnhMk19VRL9YgzR8Ujy/uFvzbExDd59EeIRYYvZPn/LjgCnkLnc htaDRTjM5wTNlBDA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id E6952118DD; Wed, 9 Jun 2021 11:39:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i+kXFJUxiAt8nF/gFOeGLQ15QjX96rJgGO1xp0O5Hkw=; b=EZ1sKebsm1bY5kIaC7UF9DQ1G7UTyLUSr9XMyF5zYN+EANnQpxhQm1tfdWkXRzv30znYUQ e2SXselrS/jgk97AcPczw3+LhCZJhHsIaF93qGh7JBJZJviOst6a3gCaRPF+ca4dF6YusJ Uuo9uoa4l0+l44z3JCV12cZIzfG1Rbk= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i+kXFJUxiAt8nF/gFOeGLQ15QjX96rJgGO1xp0O5Hkw=; b=43aWP7ErEcQ8m4WYdN2jnhMk19VRL9YgzR8Ujy/uFvzbExDd59EeIRYYvZPn/LjgCnkLnc htaDRTjM5wTNlBDA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id 4CbVN3iowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:36 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 28/34] mm, slab: make flush_slab() possible to call with irqs enabled Date: Wed, 9 Jun 2021 13:38:57 +0200 Message-Id: <20210609113903.1421-29-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=EZ1sKebs; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=43aWP7Er; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=EZ1sKebs; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=43aWP7Er; spf=pass (imf22.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: C080CC00CBD7 X-Stat-Signature: icwgdky6y3ukp8p5ffa5pthsobyiyajz X-HE-Tag: 1623238773-780982 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Currently flush_slab() is always called with disabled IRQs if it's needed, but the following patches will change that, so add a parameter to control IRQ disabling within the function, which only protects the kmem_cache_cpu manipulation and not the call to deactivate_slab() which doesn't need it. Signed-off-by: Vlastimil Babka --- mm/slub.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 12d35d6a6eaa..5c70fc17e9be 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2438,16 +2438,28 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) #endif /* CONFIG_SLUB_CPU_PARTIAL */ } -static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c) +static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c, + bool lock) { - void *freelist = c->freelist; - struct page *page = c->page; + unsigned long flags; + void *freelist; + struct page *page; + + if (lock) + local_irq_save(flags); + + freelist = c->freelist; + page = c->page; c->page = NULL; c->freelist = NULL; c->tid = next_tid(c->tid); - deactivate_slab(s, page, freelist); + if (lock) + local_irq_restore(flags); + + if (page) + deactivate_slab(s, page, freelist); stat(s, CPUSLAB_FLUSH); } @@ -2457,7 +2469,7 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu); if (c->page) - flush_slab(s, c); + flush_slab(s, c, false); unfreeze_partials_cpu(s, c); } @@ -2473,7 +2485,7 @@ static void flush_cpu_slab(void *d) struct kmem_cache_cpu *c = this_cpu_ptr(s->cpu_slab); if (c->page) - flush_slab(s, c); + flush_slab(s, c, false); unfreeze_partials(s); } From patchwork Wed Jun 9 11:38:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D16EC48BD1 for ; Wed, 9 Jun 2021 11:40:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C6ED9613BA for ; Wed, 9 Jun 2021 11:40:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C6ED9613BA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B215C6B009D; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF6AE6B009F; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 924376B009D; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0115.hostedemail.com [216.40.44.115]) by kanga.kvack.org (Postfix) with ESMTP id 4503C6B0099 for ; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B5596B9E9 for ; Wed, 9 Jun 2021 11:39:38 +0000 (UTC) X-FDA: 78233990436.02.3D20D9B Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf26.hostedemail.com (Postfix) with ESMTP id F1D994202A14 for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 58E50219ED; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qfMFwj7ppgCWhaxJh1K1eYLeFIE8J8e6h1Q7xkpTECo=; b=0PD5fXKpoeabe6dUZOIO/Z/5nY6DrOvynM1KuMFVcTyoJfmZ8VGXgpLmhe4uBjV11+D5em 4Hxy+e67kwQOxuAoWYePnfUFacJgkkdyG+VJPEIh0ekY5MhFCNIPa6MajEgBPjam6iOh6S n3okn6meKDgqv213Jv6q0qEvxYG3nFw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qfMFwj7ppgCWhaxJh1K1eYLeFIE8J8e6h1Q7xkpTECo=; b=aCiRQyu4/Yyi+HLIruMVy41bQEpCaA8+Fqp5qjFR1SOVLyXuGg55cZl3duhSMQMD/KUMJx ORaRnRU599yma8AA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 274CC11A98; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qfMFwj7ppgCWhaxJh1K1eYLeFIE8J8e6h1Q7xkpTECo=; b=0PD5fXKpoeabe6dUZOIO/Z/5nY6DrOvynM1KuMFVcTyoJfmZ8VGXgpLmhe4uBjV11+D5em 4Hxy+e67kwQOxuAoWYePnfUFacJgkkdyG+VJPEIh0ekY5MhFCNIPa6MajEgBPjam6iOh6S n3okn6meKDgqv213Jv6q0qEvxYG3nFw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qfMFwj7ppgCWhaxJh1K1eYLeFIE8J8e6h1Q7xkpTECo=; b=aCiRQyu4/Yyi+HLIruMVy41bQEpCaA8+Fqp5qjFR1SOVLyXuGg55cZl3duhSMQMD/KUMJx ORaRnRU599yma8AA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id KJweCXmowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:37 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 29/34] mm: slub: Move flush_cpu_slab() invocations __free_slab() invocations out of IRQ context Date: Wed, 9 Jun 2021 13:38:58 +0200 Message-Id: <20210609113903.1421-30-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=0PD5fXKp; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=aCiRQyu4; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=0PD5fXKp; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=aCiRQyu4; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: pkrr3bw5ycmdnkm47z898zcqoytwsef7 X-Rspamd-Queue-Id: F1D994202A14 X-HE-Tag: 1623238775-61806 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Sebastian Andrzej Siewior flush_all() flushes a specific SLAB cache on each CPU (where the cache is present). The deactivate_slab()/__free_slab() invocation happens within IPI handler and is problematic for PREEMPT_RT. The flush operation is not a frequent operation or a hot path. The per-CPU flush operation can be moved to within a workqueue. [vbabka@suse.cz: adapt to new SLUB changes] Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Vlastimil Babka --- mm/slub.c | 56 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 48 insertions(+), 8 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 5c70fc17e9be..6e747e0d7dcd 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -2474,33 +2474,73 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) unfreeze_partials_cpu(s, c); } +struct slub_flush_work { + struct work_struct work; + struct kmem_cache *s; + bool skip; +}; + /* * Flush cpu slab. * - * Called from IPI handler with interrupts disabled. + * Called from CPU work handler with migration disabled. */ -static void flush_cpu_slab(void *d) +static void flush_cpu_slab(struct work_struct *w) { - struct kmem_cache *s = d; - struct kmem_cache_cpu *c = this_cpu_ptr(s->cpu_slab); + struct kmem_cache *s; + struct kmem_cache_cpu *c; + struct slub_flush_work *sfw; + + sfw = container_of(w, struct slub_flush_work, work); + + s = sfw->s; + c = this_cpu_ptr(s->cpu_slab); if (c->page) - flush_slab(s, c, false); + flush_slab(s, c, true); unfreeze_partials(s); } -static bool has_cpu_slab(int cpu, void *info) +static bool has_cpu_slab(int cpu, struct kmem_cache *s) { - struct kmem_cache *s = info; struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu); return c->page || slub_percpu_partial(c); } +static DEFINE_MUTEX(flush_lock); +static DEFINE_PER_CPU(struct slub_flush_work, slub_flush); + static void flush_all(struct kmem_cache *s) { - on_each_cpu_cond(has_cpu_slab, flush_cpu_slab, s, 1); + struct slub_flush_work *sfw; + unsigned int cpu; + + cpus_read_lock(); + mutex_lock(&flush_lock); + + for_each_online_cpu(cpu) { + sfw = &per_cpu(slub_flush, cpu); + if (!has_cpu_slab(cpu, s)) { + sfw->skip = true; + continue; + } + INIT_WORK(&sfw->work, flush_cpu_slab); + sfw->skip = false; + sfw->s = s; + schedule_work_on(cpu, &sfw->work); + } + + for_each_online_cpu(cpu) { + sfw = &per_cpu(slub_flush, cpu); + if (sfw->skip) + continue; + flush_work(&sfw->work); + } + + mutex_unlock(&flush_lock); + cpus_read_unlock(); } /* From patchwork Wed Jun 9 11:38:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DB15C48BDF for ; Wed, 9 Jun 2021 11:46:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F3A456100A for ; Wed, 9 Jun 2021 11:46:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F3A456100A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8CCAA6B0036; Wed, 9 Jun 2021 07:46:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 87CF16B006E; Wed, 9 Jun 2021 07:46:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CFEE6B0070; Wed, 9 Jun 2021 07:46:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0018.hostedemail.com [216.40.44.18]) by kanga.kvack.org (Postfix) with ESMTP id 3A27C6B0036 for ; Wed, 9 Jun 2021 07:46:49 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D094B8249980 for ; Wed, 9 Jun 2021 11:46:48 +0000 (UTC) X-FDA: 78234008496.14.EDF3F62 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf28.hostedemail.com (Postfix) with ESMTP id 6CBB2200107C for ; Wed, 9 Jun 2021 11:46:44 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 8A313219EE; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2JMzciOjrM3f4PogJGc8S9HpllJu3Zah47/BJwa+Z4I=; b=HYimdiD0q/CIzWg6N9+ODMNUwd1vmK+xrYk38BNgyoP2eyp47gLB4RUh4/zYQD6PlFUxA5 LesyH/UW9lDttTY/jfpkBfcrWgskHewsYMPdpsNFOwWXZhmbK+9pBhc7JeD8meX6A5lLcA JSK8Xngjf3KQesVaKqF7iL7BWvNBKQ0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2JMzciOjrM3f4PogJGc8S9HpllJu3Zah47/BJwa+Z4I=; b=TikwQrejxZogM6sx8B9yhQLZFAqtzGFd6WvhKx8ugT1es4FCY/FZ93O8YnSXGrvUHti8CX 1DZ8SCb1IKNlXhCA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 5902A118DD; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2JMzciOjrM3f4PogJGc8S9HpllJu3Zah47/BJwa+Z4I=; b=HYimdiD0q/CIzWg6N9+ODMNUwd1vmK+xrYk38BNgyoP2eyp47gLB4RUh4/zYQD6PlFUxA5 LesyH/UW9lDttTY/jfpkBfcrWgskHewsYMPdpsNFOwWXZhmbK+9pBhc7JeD8meX6A5lLcA JSK8Xngjf3KQesVaKqF7iL7BWvNBKQ0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2JMzciOjrM3f4PogJGc8S9HpllJu3Zah47/BJwa+Z4I=; b=TikwQrejxZogM6sx8B9yhQLZFAqtzGFd6WvhKx8ugT1es4FCY/FZ93O8YnSXGrvUHti8CX 1DZ8SCb1IKNlXhCA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id 8GQ7FXmowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:37 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 30/34] mm: slub: Make object_map_lock a raw_spinlock_t Date: Wed, 9 Jun 2021 13:38:59 +0200 Message-Id: <20210609113903.1421-31-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=HYimdiD0; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=TikwQrej; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=HYimdiD0; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=TikwQrej; dmarc=none; spf=pass (imf28.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Rspamd-Server: rspam02 X-Stat-Signature: nsceaz9ref39c5puh8prg3b16rum4pch X-Rspamd-Queue-Id: 6CBB2200107C X-HE-Tag: 1623239204-761553 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Sebastian Andrzej Siewior The variable object_map is protected by object_map_lock. The lock is always acquired in debug code and within already atomic context Make object_map_lock a raw_spinlock_t. Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Vlastimil Babka --- mm/slub.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 6e747e0d7dcd..cfd5a7660375 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -432,7 +432,7 @@ static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page, #ifdef CONFIG_SLUB_DEBUG static unsigned long object_map[BITS_TO_LONGS(MAX_OBJS_PER_PAGE)]; -static DEFINE_SPINLOCK(object_map_lock); +static DEFINE_RAW_SPINLOCK(object_map_lock); static void __fill_map(unsigned long *obj_map, struct kmem_cache *s, struct page *page) @@ -457,7 +457,7 @@ static unsigned long *get_map(struct kmem_cache *s, struct page *page) { VM_BUG_ON(!irqs_disabled()); - spin_lock(&object_map_lock); + raw_spin_lock(&object_map_lock); __fill_map(object_map, s, page); @@ -467,7 +467,7 @@ static unsigned long *get_map(struct kmem_cache *s, struct page *page) static void put_map(unsigned long *map) __releases(&object_map_lock) { VM_BUG_ON(map != object_map); - spin_unlock(&object_map_lock); + raw_spin_unlock(&object_map_lock); } static inline unsigned int size_from_object(struct kmem_cache *s) From patchwork Wed Jun 9 11:39:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9110CC48BCF for ; Wed, 9 Jun 2021 11:40:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 489C160FD8 for ; Wed, 9 Jun 2021 11:40:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 489C160FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 37F396B0099; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 355396B009C; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E78D76B00A0; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0084.hostedemail.com [216.40.44.84]) by kanga.kvack.org (Postfix) with ESMTP id A9ACB6B009C for ; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 552FE381E for ; Wed, 9 Jun 2021 11:39:39 +0000 (UTC) X-FDA: 78233990478.02.6EA8F49 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf21.hostedemail.com (Postfix) with ESMTP id 46E4BE000255 for ; Wed, 9 Jun 2021 11:39:33 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B9ABE219EF; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YFfv9gYBIZkYhNj/PZ5egAELChMxeu671JHTmH9zWkc=; b=GGaUqfxwVSalM5ZguMtRJiHeQlalVkpBzXOaK6Qllk204MGMoM38zdwJ3taDQjy2m5hVSN eQmPidgMgA39L4chluB/i8a80PrtqXFIF7b+KRIewsrKdkQY0DJ/FK7dd/QaPZrKOuuG0M BLiOxqqcoHsy/+c98klTRE3PY2GBRek= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YFfv9gYBIZkYhNj/PZ5egAELChMxeu671JHTmH9zWkc=; b=8jqAorfI47DCQxUHvRbKOyNIQBqCTSAg0whUvcnu2IT8aECFfYaU0gkabMZMgNIPYiQRwH 1Ph+kO0oCYi2PgAA== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 8A2B311A98; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YFfv9gYBIZkYhNj/PZ5egAELChMxeu671JHTmH9zWkc=; b=GGaUqfxwVSalM5ZguMtRJiHeQlalVkpBzXOaK6Qllk204MGMoM38zdwJ3taDQjy2m5hVSN eQmPidgMgA39L4chluB/i8a80PrtqXFIF7b+KRIewsrKdkQY0DJ/FK7dd/QaPZrKOuuG0M BLiOxqqcoHsy/+c98klTRE3PY2GBRek= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YFfv9gYBIZkYhNj/PZ5egAELChMxeu671JHTmH9zWkc=; b=8jqAorfI47DCQxUHvRbKOyNIQBqCTSAg0whUvcnu2IT8aECFfYaU0gkabMZMgNIPYiQRwH 1Ph+kO0oCYi2PgAA== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id 4EVFIXmowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:37 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 31/34] mm, slub: optionally save/restore irqs in slab_[un]lock()/ Date: Wed, 9 Jun 2021 13:39:00 +0200 Message-Id: <20210609113903.1421-32-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=GGaUqfxw; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=8jqAorfI; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=GGaUqfxw; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=8jqAorfI; spf=pass (imf21.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 46E4BE000255 X-Stat-Signature: e7wms865jbsqr55s3bpsh83g69wa8kbo X-HE-Tag: 1623238773-2406 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For PREEMPT_RT we will need to disable irqs for this bit spinlock. As a preparation, add a flags parameter, and an internal version that takes additional bool parameter to control irq saving/restoring (the flags parameter is compile-time unused if the bool is a constant false). Convert ___cmpxchg_double_slab(), which also comes with the same bool parameter, to use the internal version. Signed-off-by: Vlastimil Babka --- mm/slub.c | 49 +++++++++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 18 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index cfd5a7660375..6721169f816d 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -353,18 +353,35 @@ static inline unsigned int oo_objects(struct kmem_cache_order_objects x) /* * Per slab locking using the pagelock */ -static __always_inline void slab_lock(struct page *page) +static __always_inline void +__slab_lock(struct page *page, unsigned long *flags, bool disable_irqs) { VM_BUG_ON_PAGE(PageTail(page), page); + if (disable_irqs) + local_irq_save(*flags); bit_spin_lock(PG_locked, &page->flags); } -static __always_inline void slab_unlock(struct page *page) +static __always_inline void +__slab_unlock(struct page *page, unsigned long *flags, bool disable_irqs) { VM_BUG_ON_PAGE(PageTail(page), page); + if (disable_irqs) + local_irq_restore(*flags); __bit_spin_unlock(PG_locked, &page->flags); } +static __always_inline void +slab_lock(struct page *page, unsigned long *flags) +{ + __slab_lock(page, flags, false); +} + +static __always_inline void slab_unlock(struct page *page, unsigned long *flags) +{ + __slab_unlock(page, flags, false); +} + static inline bool ___cmpxchg_double_slab(struct kmem_cache *s, struct page *page, void *freelist_old, unsigned long counters_old, void *freelist_new, unsigned long counters_new, @@ -384,21 +401,15 @@ static inline bool ___cmpxchg_double_slab(struct kmem_cache *s, struct page *pag { unsigned long flags; - if (disable_irqs) - local_irq_save(flags); - slab_lock(page); + __slab_lock(page, &flags, disable_irqs); if (page->freelist == freelist_old && page->counters == counters_old) { page->freelist = freelist_new; page->counters = counters_new; - slab_unlock(page); - if (disable_irqs) - local_irq_restore(flags); + __slab_unlock(page, &flags, disable_irqs); return true; } - slab_unlock(page); - if (disable_irqs) - local_irq_restore(flags); + __slab_unlock(page, &flags, disable_irqs); } cpu_relax(); @@ -1214,11 +1225,11 @@ static noinline int free_debug_processing( struct kmem_cache_node *n = get_node(s, page_to_nid(page)); void *object = head; int cnt = 0; - unsigned long flags; + unsigned long flags, flags2; int ret = 0; spin_lock_irqsave(&n->list_lock, flags); - slab_lock(page); + slab_lock(page, &flags2); if (s->flags & SLAB_CONSISTENCY_CHECKS) { if (!check_slab(s, page)) @@ -1251,7 +1262,7 @@ static noinline int free_debug_processing( slab_err(s, page, "Bulk freelist count(%d) invalid(%d)\n", bulk_cnt, cnt); - slab_unlock(page); + slab_unlock(page, &flags2); spin_unlock_irqrestore(&n->list_lock, flags); if (!ret) slab_fix(s, "Object at 0x%p not freed", object); @@ -4007,9 +4018,10 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page, void *addr = page_address(page); unsigned long *map; void *p; + unsigned long flags; slab_err(s, page, text, s->name); - slab_lock(page); + slab_lock(page, &flags); map = get_map(s, page); for_each_object(p, s, addr, page->objects) { @@ -4020,7 +4032,7 @@ static void list_slab_objects(struct kmem_cache *s, struct page *page, } } put_map(map); - slab_unlock(page); + slab_unlock(page, &flags); #endif } @@ -4736,8 +4748,9 @@ static void validate_slab(struct kmem_cache *s, struct page *page, { void *p; void *addr = page_address(page); + unsigned long flags; - slab_lock(page); + slab_lock(page, &flags); if (!check_slab(s, page) || !on_freelist(s, page, NULL)) goto unlock; @@ -4752,7 +4765,7 @@ static void validate_slab(struct kmem_cache *s, struct page *page, break; } unlock: - slab_unlock(page); + slab_unlock(page, &flags); } static int validate_slab_node(struct kmem_cache *s, From patchwork Wed Jun 9 11:39:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B666DC48BCF for ; Wed, 9 Jun 2021 11:40:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6049160FD8 for ; Wed, 9 Jun 2021 11:40:33 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6049160FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 623A76B009C; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 442A46B00A0; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3ECD6B009F; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id A4D4E6B0099 for ; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 458B7824999B for ; Wed, 9 Jun 2021 11:39:39 +0000 (UTC) X-FDA: 78233990478.09.D03C988 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf06.hostedemail.com (Postfix) with ESMTP id 1CAECC005A20 for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id EBBAB219F0; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kIsSOg2ibGFfBt24mwRJ/OtudI5KuCKn5sZ6lfdE+UY=; b=SAXljtSeuHX/AHMDvIbkcGQIAoE8DuboaCglNFU8aJf5EpSJzDrROkRuBnPLJy7vOhJ6gC sUb5CPMdHaJHJxg9Dsq6ASLRETL+axkYEZ3k8/WQy4WRnX8WdgEhGtk3Q6lLc+6jBIii5Q ImYH3mQlApaHg/KxsPLT110QSUOndMg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kIsSOg2ibGFfBt24mwRJ/OtudI5KuCKn5sZ6lfdE+UY=; b=CyW5d6D60EvIJEIiToc1ZOBwDgnQZJRRu1bZVUD3cI1+z8myUsHQ/SBge2sezlGaCXJHdJ kzPZ4ApGO9sHxeCg== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id BD017118DD; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kIsSOg2ibGFfBt24mwRJ/OtudI5KuCKn5sZ6lfdE+UY=; b=SAXljtSeuHX/AHMDvIbkcGQIAoE8DuboaCglNFU8aJf5EpSJzDrROkRuBnPLJy7vOhJ6gC sUb5CPMdHaJHJxg9Dsq6ASLRETL+axkYEZ3k8/WQy4WRnX8WdgEhGtk3Q6lLc+6jBIii5Q ImYH3mQlApaHg/KxsPLT110QSUOndMg= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238777; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kIsSOg2ibGFfBt24mwRJ/OtudI5KuCKn5sZ6lfdE+UY=; b=CyW5d6D60EvIJEIiToc1ZOBwDgnQZJRRu1bZVUD3cI1+z8myUsHQ/SBge2sezlGaCXJHdJ kzPZ4ApGO9sHxeCg== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id CKiuLXmowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:37 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 32/34] mm, slub: make slab_lock() disable irqs with PREEMPT_RT Date: Wed, 9 Jun 2021 13:39:01 +0200 Message-Id: <20210609113903.1421-33-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 1CAECC005A20 Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=SAXljtSe; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=CyW5d6D6; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=SAXljtSe; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=CyW5d6D6; spf=pass (imf06.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Stat-Signature: yw1qu13roewhbst6bzc475pbmoai9s1w X-HE-Tag: 1623238775-401471 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We need to disable irqs around slab_lock() (a bit spinlock) to make it irq-safe. The calls to slab_lock() are nested under spin_lock_irqsave() which doesn't disable irqs on PREEMPT_RT, so add explicit disabling with PREEMPT_RT. We also distinguish cmpxchg_double_slab() where we do the disabling explicitly and __cmpxchg_double_slab() for contexts with already disabled irqs. However these context are also typically spin_lock_irqsave() thus insufficient on PREEMPT_RT. Thus, change __cmpxchg_double_slab() to be same as cmpxchg_double_slab() on PREEMPT_RT. Signed-off-by: Vlastimil Babka --- mm/slub.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 6721169f816d..12e966f07f19 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -374,12 +374,12 @@ __slab_unlock(struct page *page, unsigned long *flags, bool disable_irqs) static __always_inline void slab_lock(struct page *page, unsigned long *flags) { - __slab_lock(page, flags, false); + __slab_lock(page, flags, IS_ENABLED(CONFIG_PREEMPT_RT)); } static __always_inline void slab_unlock(struct page *page, unsigned long *flags) { - __slab_unlock(page, flags, false); + __slab_unlock(page, flags, IS_ENABLED(CONFIG_PREEMPT_RT)); } static inline bool ___cmpxchg_double_slab(struct kmem_cache *s, struct page *page, @@ -422,14 +422,19 @@ static inline bool ___cmpxchg_double_slab(struct kmem_cache *s, struct page *pag return false; } -/* Interrupts must be disabled (for the fallback code to work right) */ +/* + * Interrupts must be disabled (for the fallback code to work right), typically + * by an _irqsave() lock variant. Except on PREEMPT_RT where locks are different + * so we disable interrupts explicitly here. + */ static inline bool __cmpxchg_double_slab(struct kmem_cache *s, struct page *page, void *freelist_old, unsigned long counters_old, void *freelist_new, unsigned long counters_new, const char *n) { return ___cmpxchg_double_slab(s, page, freelist_old, counters_old, - freelist_new, counters_new, n, false); + freelist_new, counters_new, n, + IS_ENABLED(CONFIG_PREEMPT_RT)); } static inline bool cmpxchg_double_slab(struct kmem_cache *s, struct page *page, From patchwork Wed Jun 9 11:39:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A588FC48BCD for ; Wed, 9 Jun 2021 11:40:35 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5B20E60FD8 for ; Wed, 9 Jun 2021 11:40:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5B20E60FD8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7FCCA6B009F; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EFB66B00A2; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E43D6B00A1; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id CEF346B009E for ; Wed, 9 Jun 2021 07:39:39 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 75BD1180AD81A for ; Wed, 9 Jun 2021 11:39:39 +0000 (UTC) X-FDA: 78233990478.29.D79DFAC Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf19.hostedemail.com (Postfix) with ESMTP id 7A5009001460 for ; Wed, 9 Jun 2021 11:39:34 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 2C545219F2; Wed, 9 Jun 2021 11:39:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238778; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HCgR6cFavL9oqyY6iyzsui86wbbyKVk3bDILLJY4HV8=; b=crODmQTDxbTwm6t6lBpB0byfYSMxf74yooLAtc8lHbpb02RPkSiJUlnPHtU0eqQWMH+Lf8 VvwMsDiBPjK1FO9Nd/XM2vLQf6QQsEKM+p4CC1qINFqFVrM0Iw1NyZbmmQhRyaVsodYED1 o6XOuU7FkM8BcM/XBWb//mhRRCdLZtE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238778; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HCgR6cFavL9oqyY6iyzsui86wbbyKVk3bDILLJY4HV8=; b=qm804xEyxAArw/n+EZ//Bk8Fdtsa5qLcfGQx11svOjsk97fnIHk8vBEA7nQJAAl6JCYA7I q8Epun7nFiweTgCw== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id ED49B11A98; Wed, 9 Jun 2021 11:39:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238778; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HCgR6cFavL9oqyY6iyzsui86wbbyKVk3bDILLJY4HV8=; b=crODmQTDxbTwm6t6lBpB0byfYSMxf74yooLAtc8lHbpb02RPkSiJUlnPHtU0eqQWMH+Lf8 VvwMsDiBPjK1FO9Nd/XM2vLQf6QQsEKM+p4CC1qINFqFVrM0Iw1NyZbmmQhRyaVsodYED1 o6XOuU7FkM8BcM/XBWb//mhRRCdLZtE= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238778; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HCgR6cFavL9oqyY6iyzsui86wbbyKVk3bDILLJY4HV8=; b=qm804xEyxAArw/n+EZ//Bk8Fdtsa5qLcfGQx11svOjsk97fnIHk8vBEA7nQJAAl6JCYA7I q8Epun7nFiweTgCw== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id GC5sOXmowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:37 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 33/34] mm, slub: use migrate_disable() on PREEMPT_RT Date: Wed, 9 Jun 2021 13:39:02 +0200 Message-Id: <20210609113903.1421-34-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=crODmQTD; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=qm804xEy; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=crODmQTD; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=qm804xEy; dmarc=none; spf=pass (imf19.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Stat-Signature: 7odfkpxr8jb8pm5gumhk4krrinezinsa X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 7A5009001460 X-HE-Tag: 1623238774-566153 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We currently use preempt_disable() (directly or via get_cpu_ptr()) to stabilize the pointer to kmem_cache_cpu. On PREEMPT_RT this would be incompatible with the list_lock spinlock. We can use migrate_disable() instead, but that increases overhead on !PREEMPT_RT as it's an unconditional function call even though it's ultimately a migrate_disable() there. In order to get the best available mechanism on both PREEMPT_RT and !PREEMPT_RT, introduce private slub_get_cpu_ptr() and slub_put_cpu_ptr() wrappers and use them. Signed-off-by: Vlastimil Babka Signed-off-by: Vlastimil Babka --- mm/slub.c | 41 +++++++++++++++++++++++++++++++---------- 1 file changed, 31 insertions(+), 10 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 12e966f07f19..caa206213e72 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -115,6 +115,26 @@ * the fast path and disables lockless freelists. */ +/* + * We could simply use migrate_disable()/enable() but as long as it's a + * function call even on !PREEMPT_RT, use inline preempt_disable() there. + */ +#ifdef CONFIG_PREEMPT_RT +#define slub_get_cpu_ptr(var) get_cpu_ptr(var) +#define slub_put_cpu_ptr(var) put_cpu_ptr(var) +#else +#define slub_get_cpu_ptr(var) \ +({ \ + migrate_disable(); \ + this_cpu_ptr(var); \ +}) +#define slub_put_cpu_ptr(var) \ +do { \ + (void)(var); \ + migrate_enable(); \ +} while (0) +#endif + #ifdef CONFIG_SLUB_DEBUG #ifdef CONFIG_SLUB_DEBUG_ON DEFINE_STATIC_KEY_TRUE(slub_debug_enabled); @@ -2419,7 +2439,7 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) int pages; int pobjects; - preempt_disable(); + slub_get_cpu_ptr(s->cpu_slab); do { pages = 0; pobjects = 0; @@ -2450,7 +2470,7 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain) } while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page) != oldpage); - preempt_enable(); + slub_put_cpu_ptr(s->cpu_slab); #endif /* CONFIG_SLUB_CPU_PARTIAL */ } @@ -2759,7 +2779,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (unlikely(!pfmemalloc_match(page, gfpflags))) goto deactivate_slab; - /* must check again c->page in case IRQ handler changed it */ + /* must check again c->page in case we got preempted and it changed */ local_irq_save(flags); if (unlikely(page != c->page)) { local_irq_restore(flags); @@ -2818,7 +2838,8 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, } if (unlikely(!slub_percpu_partial(c))) { local_irq_restore(flags); - goto new_objects; /* stolen by an IRQ handler */ + /* we were preempted and partial list got empty */ + goto new_objects; } page = c->page = slub_percpu_partial(c); @@ -2834,9 +2855,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (freelist) goto check_new_page; - put_cpu_ptr(s->cpu_slab); + slub_put_cpu_ptr(s->cpu_slab); page = new_slab(s, gfpflags, node); - c = get_cpu_ptr(s->cpu_slab); + c = slub_get_cpu_ptr(s->cpu_slab); if (unlikely(!page)) { slab_out_of_memory(s, gfpflags, node); @@ -2919,12 +2940,12 @@ static void *__slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, * cpu before disabling preemption. Need to reload cpu area * pointer. */ - c = get_cpu_ptr(s->cpu_slab); + c = slub_get_cpu_ptr(s->cpu_slab); #endif p = ___slab_alloc(s, gfpflags, node, addr, c); #ifdef CONFIG_PREEMPTION - put_cpu_ptr(s->cpu_slab); + slub_put_cpu_ptr(s->cpu_slab); #endif return p; } @@ -3445,7 +3466,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, * IRQs, which protects against PREEMPT and interrupts * handlers invoking normal fastpath. */ - c = get_cpu_ptr(s->cpu_slab); + c = slub_get_cpu_ptr(s->cpu_slab); local_irq_disable(); for (i = 0; i < size; i++) { @@ -3491,7 +3512,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, } c->tid = next_tid(c->tid); local_irq_enable(); - put_cpu_ptr(s->cpu_slab); + slub_put_cpu_ptr(s->cpu_slab); /* * memcg and kmem_cache debug support and memory initialization. From patchwork Wed Jun 9 11:39:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 12309895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 716F2C48BCD for ; Wed, 9 Jun 2021 11:42:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id F35546100B for ; Wed, 9 Jun 2021 11:42:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F35546100B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B1FB26B009E; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF78B6B00A0; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D5786B00A1; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0169.hostedemail.com [216.40.44.169]) by kanga.kvack.org (Postfix) with ESMTP id 4C9126B009E for ; Wed, 9 Jun 2021 07:39:40 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E1BF1181AEF2A for ; Wed, 9 Jun 2021 11:39:39 +0000 (UTC) X-FDA: 78233990478.01.27685CE Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf08.hostedemail.com (Postfix) with ESMTP id 769378019361 for ; Wed, 9 Jun 2021 11:39:35 +0000 (UTC) Received: from imap.suse.de (imap-alt.suse-dmz.suse.de [192.168.254.47]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 5FB53219F3; Wed, 9 Jun 2021 11:39:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238778; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=biJeHjWSGncOZGlXHpbWSSTom99Ta0fP7Xuy9hrQoQs=; b=EPKM526Ob5SuuPwfwS4F/f7C9Jstx0dXTJ8MK39PR3Mg/n7TcD3q2Ygij/aBrlpHjMWPfR bwSQUT/pyFYe7AesqoDDQgFGJfblBBfAMKXna9soluHJLpWVAfxYM1vRjZIKW/afUyp66M rbt20uGgdGOSUsA7POqe2ml/9eriMbU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238778; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=biJeHjWSGncOZGlXHpbWSSTom99Ta0fP7Xuy9hrQoQs=; b=a70dpne6qRpiupTpqhhti+0l4x+X2Ly2sLyMmy3KYG12xfZyp0tA78wHlj0w5VxSfWWl51 p407KyPv4ZwbRODg== Received: from imap3-int (imap-alt.suse-dmz.suse.de [192.168.254.47]) by imap.suse.de (Postfix) with ESMTP id 2D2E1118DD; Wed, 9 Jun 2021 11:39:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1623238778; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=biJeHjWSGncOZGlXHpbWSSTom99Ta0fP7Xuy9hrQoQs=; b=EPKM526Ob5SuuPwfwS4F/f7C9Jstx0dXTJ8MK39PR3Mg/n7TcD3q2Ygij/aBrlpHjMWPfR bwSQUT/pyFYe7AesqoDDQgFGJfblBBfAMKXna9soluHJLpWVAfxYM1vRjZIKW/afUyp66M rbt20uGgdGOSUsA7POqe2ml/9eriMbU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1623238778; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=biJeHjWSGncOZGlXHpbWSSTom99Ta0fP7Xuy9hrQoQs=; b=a70dpne6qRpiupTpqhhti+0l4x+X2Ly2sLyMmy3KYG12xfZyp0tA78wHlj0w5VxSfWWl51 p407KyPv4ZwbRODg== Received: from director2.suse.de ([192.168.254.72]) by imap3-int with ESMTPSA id 4L+RCnqowGD6XgAALh3uQQ (envelope-from ); Wed, 09 Jun 2021 11:39:38 +0000 From: Vlastimil Babka To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Christoph Lameter , David Rientjes , Pekka Enberg , Joonsoo Kim Cc: Sebastian Andrzej Siewior , Thomas Gleixner , Mel Gorman , Jesper Dangaard Brouer , Peter Zijlstra , Jann Horn , Vlastimil Babka Subject: [RFC v2 34/34] mm, slub: convert kmem_cpu_slab protection to local_lock Date: Wed, 9 Jun 2021 13:39:03 +0200 Message-Id: <20210609113903.1421-35-vbabka@suse.cz> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210609113903.1421-1-vbabka@suse.cz> References: <20210609113903.1421-1-vbabka@suse.cz> MIME-Version: 1.0 Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=EPKM526O; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=a70dpne6; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=EPKM526O; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=a70dpne6; spf=pass (imf08.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 769378019361 X-Stat-Signature: 6rhge1ac776zjduc9juew7ngr6ie3jwh X-HE-Tag: 1623238775-455270 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Embed local_lock into struct kmem_cpu_slab and use the irq-safe versions of local_lock instead of plain local_irq_save/restore. On !PREEMPT_RT that's equivalent, with better lockdep visibility. On PREEMPT_RT that means better preemption. However, the cost on PREEMPT_RT is the loss of lockless fast paths which only work with cpu freelist. Those are designed to detect and recover from being preempted by other conflicting operations (both fast or slow path), but the slow path operations assume they cannot be preempted by a fast path operation, which is guaranteed naturally with disabled irqs. With local locks on PREEMPT_RT, the fast paths now also need to take the local lock to avoid races. In the allocation fastpath slab_alloc_node() we can just defer to the slowpath __slab_alloc() which also works with cpu freelist, but under the local lock. In the free fastpath do_slab_free() we have to add a new local lock protected version of freeing to the cpu freelist, as the existing slowpath only works with the page freelist. Also update the comment about locking scheme in SLUB to reflect changes done by this series. Signed-off-by: Vlastimil Babka --- include/linux/slub_def.h | 2 + mm/slub.c | 138 ++++++++++++++++++++++++++++++--------- 2 files changed, 110 insertions(+), 30 deletions(-) diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index dcde82a4434c..b5bcac29b979 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -10,6 +10,7 @@ #include #include #include +#include enum stat_item { ALLOC_FASTPATH, /* Allocation from cpu slab */ @@ -41,6 +42,7 @@ enum stat_item { NR_SLUB_STAT_ITEMS }; struct kmem_cache_cpu { + local_lock_t lock; /* Protects the fields below except stat */ void **freelist; /* Pointer to next available object */ unsigned long tid; /* Globally unique transaction id */ struct page *page; /* The slab from which we are allocating */ diff --git a/mm/slub.c b/mm/slub.c index caa206213e72..500720ec1e57 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -43,13 +43,22 @@ /* * Lock order: * 1. slab_mutex (Global Mutex) - * 2. node->list_lock - * 3. slab_lock(page) (Only on some arches and for debugging) + * 2. node->list_lock (Spinlock) + * OR + * kmem_cache->cpu_slab->lock (Local lock) + * 3. slab_lock(page) (Only on some arches or for debugging) + * 4. object_map_lock (Only for debugging) * * slab_mutex * * The role of the slab_mutex is to protect the list of all the slabs * and to synchronize major metadata changes to slab cache structures. + * Also synchronizes memory hotplug callbacks. + * + * slab_lock + * + * The slab_lock is a wrapper around the page lock, thus it is a bit + * spinlock. * * The slab_lock is only used for debugging and on arches that do not * have the ability to do a cmpxchg_double. It only protects: @@ -58,6 +67,8 @@ * C. page->objects -> Number of objects in page * D. page->frozen -> frozen state * + * Frozen slabs + * * If a slab is frozen then it is exempt from list management. It is not * on any list except per cpu partial list. The processor that froze the * slab is the one who can perform list operations on the page. Other @@ -65,6 +76,8 @@ * froze the slab is the only one that can retrieve the objects from the * page's freelist. * + * list_lock + * * The list_lock protects the partial and full list on each node and * the partial slab counter. If taken then no new slabs may be added or * removed from the lists nor make the number of partial slabs be modified. @@ -76,10 +89,36 @@ * slabs, operations can continue without any centralized lock. F.e. * allocating a long series of objects that fill up slabs does not require * the list lock. - * Interrupts are disabled during allocation and deallocation in order to - * make the slab allocator safe to use in the context of an irq. In addition - * interrupts are disabled to ensure that the processor does not change - * while handling per_cpu slabs, due to kernel preemption. + * + * cpu_slab->lock local lock + * + * This locks protect slowpath manipulation of all kmem_cache_cpu fields + * except the stat counters. This is a percpu structure manipulated only by + * the local cpu, so the lock protects against being preempted or interrupted + * by an irq. Fast path operations rely on lockless operations instead. + * On PREEMPT_RT, the local lock does not actually disable irqs (and thus + * prevent the lockless operations), so fastpath operations also need to take + * the lock and are no longer lockless. + * + * lockless fastpaths + * + * The fast path allocation (slab_alloc_node()) and freeing (do_slab_free()) + * are fully lockless when satisfied from the percpu slab (and when + * cmpxchg_double is possible to use, otherwise slab_lock is taken). + * They also don't disable preemption or migration or irqs. They rely on + * the transaction id (tid) field to detect being preempted or moved to + * another cpu. + * + * irq, preemption, migration considerations + * + * Interrupts are disabled as part of list_lock or local_lock operations, or + * around the slab_lock operation, in order to make the slab allocator safe + * to use in the context of an irq. + * + * In addition, preemption (or migration on PREEMPT_RT) is disabled in the + * allocation slowpath, bulk allocation, and put_cpu_partial(), so that the + * local cpu doesn't change in the process and e.g. the kmem_cache_cpu pointer + * doesn't have to be revalidated in each section protected by the local lock. * * SLUB assigns one slab for allocation to each processor. * Allocations only occur from these slabs called cpu slabs. @@ -2179,9 +2218,13 @@ static inline void note_cmpxchg_failure(const char *n, static void init_kmem_cache_cpus(struct kmem_cache *s) { int cpu; + struct kmem_cache_cpu *c; - for_each_possible_cpu(cpu) - per_cpu_ptr(s->cpu_slab, cpu)->tid = init_tid(cpu); + for_each_possible_cpu(cpu) { + c = per_cpu_ptr(s->cpu_slab, cpu); + local_lock_init(&c->lock); + c->tid = init_tid(cpu); + } } /* @@ -2482,7 +2525,7 @@ static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c, struct page *page; if (lock) - local_irq_save(flags); + local_lock_irqsave(&s->cpu_slab->lock, flags); freelist = c->freelist; page = c->page; @@ -2492,7 +2535,7 @@ static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c, c->tid = next_tid(c->tid); if (lock) - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); if (page) deactivate_slab(s, page, freelist); @@ -2780,9 +2823,9 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, goto deactivate_slab; /* must check again c->page in case we got preempted and it changed */ - local_irq_save(flags); + local_lock_irqsave(&s->cpu_slab->lock, flags); if (unlikely(page != c->page)) { - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); goto reread_page; } freelist = c->freelist; @@ -2793,7 +2836,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, if (!freelist) { c->page = NULL; - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); stat(s, DEACTIVATE_BYPASS); goto new_slab; } @@ -2802,7 +2845,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, load_freelist: - lockdep_assert_irqs_disabled(); + lockdep_assert_held(this_cpu_ptr(&s->cpu_slab->lock)); /* * freelist is pointing to the list of objects to be used. @@ -2812,39 +2855,39 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, VM_BUG_ON(!c->page->frozen); c->freelist = get_freepointer(s, freelist); c->tid = next_tid(c->tid); - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); return freelist; deactivate_slab: - local_irq_save(flags); + local_lock_irqsave(&s->cpu_slab->lock, flags); if (page != c->page) { - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); goto reread_page; } freelist = c->freelist; c->page = NULL; c->freelist = NULL; - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); deactivate_slab(s, page, freelist); new_slab: if (slub_percpu_partial(c)) { - local_irq_save(flags); + local_lock_irqsave(&s->cpu_slab->lock, flags); if (unlikely(c->page)) { - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); goto reread_page; } if (unlikely(!slub_percpu_partial(c))) { - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); /* we were preempted and partial list got empty */ goto new_objects; } page = c->page = slub_percpu_partial(c); slub_set_percpu_partial(c, page); - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); stat(s, CPU_PARTIAL_ALLOC); goto redo; } @@ -2897,7 +2940,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, retry_load_page: - local_irq_save(flags); + local_lock_irqsave(&s->cpu_slab->lock, flags); if (unlikely(c->page)) { void *flush_freelist = c->freelist; struct page *flush_page = c->page; @@ -2906,7 +2949,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, c->freelist = NULL; c->tid = next_tid(c->tid); - local_irq_restore(flags); + local_unlock_irqrestore(&s->cpu_slab->lock, flags); deactivate_slab(s, flush_page, flush_freelist); @@ -3025,7 +3068,15 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, object = c->freelist; page = c->page; - if (unlikely(!object || !page || !node_match(page, node))) { + /* + * We cannot use the lockless fastpath on PREEMPT_RT because if a + * slowpath has taken the local_lock_irqsave(), it is not protected + * against a fast path operation in an irq handler. So we need to take + * the slow path which uses local_lock. It is still relatively fast if + * there is a suitable cpu freelist. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT) || + unlikely(!object || !page || !node_match(page, node))) { object = __slab_alloc(s, gfpflags, node, addr, c); } else { void *next_object = get_freepointer_safe(s, object); @@ -3285,6 +3336,7 @@ static __always_inline void do_slab_free(struct kmem_cache *s, barrier(); if (likely(page == c->page)) { +#ifndef CONFIG_PREEMPT_RT void **freelist = READ_ONCE(c->freelist); set_freepointer(s, tail_obj, freelist); @@ -3297,6 +3349,32 @@ static __always_inline void do_slab_free(struct kmem_cache *s, note_cmpxchg_failure("slab_free", s, tid); goto redo; } +#else /* CONFIG_PREEMPT_RT */ + /* + * We cannot use the lockless fastpath on PREEMPT_RT because if + * a slowpath has taken the local_lock_irqsave(), it is not + * protected against a fast path operation in an irq handler. So + * we need to take the local_lock. We shouldn't simply defer to + * __slab_free() as that wouldn't use the cpu freelist at all. + */ + unsigned long flags; + void **freelist; + + local_lock_irqsave(&s->cpu_slab->lock, flags); + c = this_cpu_ptr(s->cpu_slab); + if (unlikely(page != c->page)) { + local_unlock_irqrestore(&s->cpu_slab->lock, flags); + goto redo; + } + tid = c->tid; + freelist = c->freelist; + + set_freepointer(s, tail_obj, freelist); + c->freelist = head; + c->tid = next_tid(tid); + + local_unlock_irqrestore(&s->cpu_slab->lock, flags); +#endif stat(s, FREE_FASTPATH); } else __slab_free(s, page, head, tail_obj, cnt, addr); @@ -3467,7 +3545,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, * handlers invoking normal fastpath. */ c = slub_get_cpu_ptr(s->cpu_slab); - local_irq_disable(); + local_lock_irq(&s->cpu_slab->lock); for (i = 0; i < size; i++) { void *object = kfence_alloc(s, s->object_size, flags); @@ -3488,7 +3566,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, */ c->tid = next_tid(c->tid); - local_irq_enable(); + local_unlock_irq(&s->cpu_slab->lock); /* * Invoking slow path likely have side-effect @@ -3502,7 +3580,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, c = this_cpu_ptr(s->cpu_slab); maybe_wipe_obj_freeptr(s, p[i]); - local_irq_disable(); + local_lock_irq(&s->cpu_slab->lock); continue; /* goto for-loop */ } @@ -3511,7 +3589,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, maybe_wipe_obj_freeptr(s, p[i]); } c->tid = next_tid(c->tid); - local_irq_enable(); + local_unlock_irq(&s->cpu_slab->lock); slub_put_cpu_ptr(s->cpu_slab); /* @@ -3522,7 +3600,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, slab_want_init_on_alloc(flags, s)); return i; error: - local_irq_enable(); + local_unlock_irq(&s->cpu_slab->lock); slab_post_alloc_hook(s, objcg, flags, i, p, false); __kmem_cache_free_bulk(s, i, p); return 0;