From patchwork Tue Jan 17 13:40:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jesper Dangaard Brouer X-Patchwork-Id: 13104639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD7DDC63797 for ; Tue, 17 Jan 2023 13:40:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4718C6B0071; Tue, 17 Jan 2023 08:40:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4219D6B0073; Tue, 17 Jan 2023 08:40:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E9AC6B0074; Tue, 17 Jan 2023 08:40:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1E2656B0071 for ; Tue, 17 Jan 2023 08:40:13 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id CB2E41A08D9 for ; Tue, 17 Jan 2023 13:40:12 +0000 (UTC) X-FDA: 80364399864.28.B7D671D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 328291C000B for ; Tue, 17 Jan 2023 13:40:10 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SxNjC6so; spf=pass (imf20.hostedemail.com: domain of brouer@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=brouer@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673962811; a=rsa-sha256; cv=none; b=4DdYliC77C5Cxn5I56HeEKfLcJKyNh8iI84stZlyNIp7T2g3ME9hzyM0J7g6sMKOVydxxR t0jJSFHAeeVSARe6CBojVCvPcit2NJQNm6V0ywhdeYSIpLEQFxRpSJg/bPB3yKxusmfeUH 9/9nmmM5F7vMmM4VcNSYEjgnh+2qJMM= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SxNjC6so; spf=pass (imf20.hostedemail.com: domain of brouer@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=brouer@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673962811; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=i9PEgqUHqyQJNAleeBEFgDDSWNqsnvJXCZKuturWNw8=; b=TM8BBma6y0arUjiw4hNpborabDXE3/szBr8Z3yBhlNOoRC/ep7AfX8kWegw5riJnAzVQ7j rFhvmpx8hqsX+Eiq7Amb0JjVKDIyf7AwMiYV8UdosePKAXE0xtEJwL+ttpvA79ZeDV/G+G pBfsH5Hp9IcWEbZDn6Oise0NmWRfjRI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673962810; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=i9PEgqUHqyQJNAleeBEFgDDSWNqsnvJXCZKuturWNw8=; b=SxNjC6soTUZ7a6AmrbH1tCqFQMU/MuBTN+UBXG0McyAJU3+VkWupqoepDglECx8v1RAqo4 JiKSKYarX7l+agZDu/4HD+qHykIK5rcf0sxFHW4v2PERGvpWIOfTHyep+GADVZYhUZBsJ7 2h9oi48pAzMaTG51X06V8Asq/SeqEbs= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-339-JsaqHhlKMWmm04iTOFCIAg-1; Tue, 17 Jan 2023 08:40:03 -0500 X-MC-Unique: JsaqHhlKMWmm04iTOFCIAg-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1BF52857F43; Tue, 17 Jan 2023 13:40:03 +0000 (UTC) Received: from firesoul.localdomain (ovpn-208-34.brq.redhat.com [10.40.208.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9185740C2064; Tue, 17 Jan 2023 13:40:01 +0000 (UTC) Received: from [192.168.42.3] (localhost [IPv6:::1]) by firesoul.localdomain (Postfix) with ESMTP id 8BA3830721A6C; Tue, 17 Jan 2023 14:40:00 +0100 (CET) Subject: [PATCH RFC] mm+net: allow to set kmem_cache create flag for SLAB_NEVER_MERGE From: Jesper Dangaard Brouer To: netdev@vger.kernel.org, linux-mm@kvack.org Cc: Jesper Dangaard Brouer , Christoph Lameter , Andrew Morton , Mel Gorman , Joonsoo Kim , penberg@kernel.org, penberg@kernel.org, vbabka@suse.cz, Jakub Kicinski , "David S. Miller" , edumazet@google.com, pabeni@redhat.com Date: Tue, 17 Jan 2023 14:40:00 +0100 Message-ID: <167396280045.539803.7540459812377220500.stgit@firesoul> User-Agent: StGit/1.4 MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Rspam-User: X-Rspamd-Queue-Id: 328291C000B X-Rspamd-Server: rspam01 X-Stat-Signature: 9d9kk7rgb1m6rfcxqd53tdxs9hhsc58e X-HE-Tag: 1673962810-127578 X-HE-Meta: U2FsdGVkX18u9jN9QYZu08UyBXgKhE5E4kmIRWIwWEte8aByt2W69WuZ3Fnql02lFBhJWdZmvzgdhLabMBAoTyFX2cO+O+6uf7EjH1ZmJb2i/UxemU3BAuXCVQ8FOYbBGjORBtKhQvX7CDCDp1NDscNYx9Xzek7k5GuAm66nQpZiDc+oluPfAfj0G8TtIpjY+syFASLk98ElmBdA48rnTpFXawy3AOl/JN9etIrpDW/NrmA7ijqEbSr8HGTc79SDURy5nuT+t6DVaVWdMcbxOMC4+2CwLtfuj0xRVesc4iw3M522zihqgrMy0GisWR8dKpMOPm/c0ZgqV/1f0y008lg2W8YnLOYO0UQxSzNiC5R67EKgQS5jrdoXMycr42hrjGJ5ZkVKrBaRxdlKp6lL1pzFTfrRJZRuboobL30e67mIRg7uSP63IPS0qcCbY/oanZXEXC99qtwRssTzk4zYqL4ru+rJ0eCpjeNCWEgV257KJUth1YTGBbmvGJdjR8jePv+lO/74u8cWrLI9ARSZUHxMgvV2y9RPrFJ8l80YWcw917kLI139k8Rt5RDbogFD6Ncne5I9PH7W01cANnonhShYwJFupqE9P2taltBanSwGVWThVz54KRdBlb7J62PE/xeLdGIQGBZHJvkOgtrRJToirx8VoWYwnuCzCDCYbjwTARIe1VXRwGIL67wSYRWHsovA4ELBgUS+vCTIe4Isq8oEb46p+CLlWbTVwuOY+96fHuw/KLNK2pV4C9O+6zcW6nyihYb4+c2HjS0F1nNozs6ZOVbSpV+OL8AL2CPC49FLtzlGtnwi6dY27DOPiOGr5vto1XbEjo3IdDuN1SIvs1bNpSmFw6E0otq9P1aTONGVOGcq/w1Q0FijYeoRdpBG4tzOMlr47ugimWAngjCFQjSNsHj0NfPgLdQbSZvbwDk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allow API users of kmem_cache_create to specify that they don't want any slab merge or aliasing (with similar sized objects). Use this in network stack and kfence_test. The SKB (sk_buff) kmem_cache slab is critical for network performance. Network stack uses kmem_cache_{alloc,free}_bulk APIs to gain performance by amortising the alloc/free cost. For the bulk API to perform efficiently the slub fragmentation need to be low. Especially for the SLUB allocator, the efficiency of bulk free API depend on objects belonging to the same slab (page). When running different network performance microbenchmarks, I started to notice that performance was reduced (slightly) when machines had longer uptimes. I believe the cause was 'skbuff_head_cache' got aliased/merged into the general slub for 256 bytes sized objects (with my kernel config, without CONFIG_HARDENED_USERCOPY). For SKB kmem_cache network stack have reasons for not merging, but it varies depending on kernel config (e.g. CONFIG_HARDENED_USERCOPY). We want to explicitly set SLAB_NEVER_MERGE for this kmem_cache. Signed-off-by: Jesper Dangaard Brouer Reported-by: David Sterba Signed-off-by: Jesper Dangaard Brouer Signed-off-by: Vlastimil Babka Acked-by: Jesper Dangaard Brouer flags & SLAB_NEVER_MERGE)) + if (slab_nomerge || (s->flags & SLAB_NEVER_MERGE_FLAGS)) return 1; if (s->ctor) @@ -173,7 +173,7 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align, size = ALIGN(size, align); flags = kmem_cache_flags(size, flags, name); - if (flags & SLAB_NEVER_MERGE) + if (flags & SLAB_NEVER_MERGE_FLAGS) return NULL; list_for_each_entry_reverse(s, &slab_caches, list) { diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 79c9e795a964..799b9914457b 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -4629,12 +4629,23 @@ static void skb_extensions_init(void) static void skb_extensions_init(void) {} #endif +/* The SKB kmem_cache slab is critical for network performance. Never + * merge/alias the slab with similar sized objects. This avoids fragmentation + * that hurts performance of kmem_cache_{alloc,free}_bulk APIs. + */ +#ifndef CONFIG_SLUB_TINY +#define FLAG_SKB_NEVER_MERGE SLAB_NEVER_MERGE +#else /* CONFIG_SLUB_TINY - simple loop in kmem_cache_alloc_bulk */ +#define FLAG_SKB_NEVER_MERGE 0 +#endif + void __init skb_init(void) { skbuff_head_cache = kmem_cache_create_usercopy("skbuff_head_cache", sizeof(struct sk_buff), 0, - SLAB_HWCACHE_ALIGN|SLAB_PANIC, + SLAB_HWCACHE_ALIGN|SLAB_PANIC| + FLAG_SKB_NEVER_MERGE, offsetof(struct sk_buff, cb), sizeof_field(struct sk_buff, cb), NULL);