From patchwork Mon Jun 18 09:18:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10470369 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7217D6032A for ; Mon, 18 Jun 2018 09:18:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 66AA928740 for ; Mon, 18 Jun 2018 09:18:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5B7722883A; Mon, 18 Jun 2018 09:18:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C10FC28740 for ; Mon, 18 Jun 2018 09:18:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CEDCE6B000A; Mon, 18 Jun 2018 05:18:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C775D6B0008; Mon, 18 Jun 2018 05:18:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B40706B0269; Mon, 18 Jun 2018 05:18:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-wr0-f198.google.com (mail-wr0-f198.google.com [209.85.128.198]) by kanga.kvack.org (Postfix) with ESMTP id 4F8726B0008 for ; Mon, 18 Jun 2018 05:18:30 -0400 (EDT) Received: by mail-wr0-f198.google.com with SMTP id x6-v6so11355677wrl.6 for ; Mon, 18 Jun 2018 02:18:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=s8kGYtwqxuQjBjt50ABAKFfnnbLYJ+Tb8OFykkxbMfI=; b=PffFbdLgC+D2UaudtDanPNTC9edhiN4RITbawZIfA1hxfDgVr3x5wG87JAnpOhLk/Q 43eWlIMj/BGjOk88Q590DKM0IlIF6SRbqHhgmoKCPeTFu3IPCJe+h/sa8Zi12xwnwWA0 4e9/2EtghvmrBGpMek6TQAOfM35vG6Uul/9+qw5MGXau/G0qsLCMiH6ugQ6UrHhC8aA+ 7eCBE45Wh3hWey3P9MgxzHrlTih7hlZJcYaIwy9c4cng6+vPK4c4z1iN/EGrxHHOv/ME +KFy3X2f4GzGTGz+SKsi63LYzVItX3h+cItnOGCm9FiO3NkclMrpSJcBGhqiZf6bbWVh O6lA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: APt69E0F8KwMBUJqKm58frA2nEsRKMVfyW7WxJulGazMGkhDRMdsndgs Ue8Ize/xYqmglqL8a963yOBedzFAmPnVYGDq+FsP7vi7uLSktHmA32hjk/nXpBAAhS7+inRRpAl vEIM7lV+8GKK1yFXLNkUhPzFsSGpvyITBbN6azRyEtQVQGpprECSoLj9VOVmec/r6RQ== X-Received: by 2002:a50:e08d:: with SMTP id f13-v6mr3096444edl.198.1529313509857; Mon, 18 Jun 2018 02:18:29 -0700 (PDT) X-Google-Smtp-Source: ADUXVKKiyqnYCd+BNjonC1oMW2ESBapEHAaFnv5utePW2jah/Or4GyiUcg9wY/voblWZnGMBd9fP X-Received: by 2002:a50:e08d:: with SMTP id f13-v6mr3096400edl.198.1529313508602; Mon, 18 Jun 2018 02:18:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1529313508; cv=none; d=google.com; s=arc-20160816; b=mUAMCoVUTFRto0xJaBbet6qdyrltrKwVlkUJKcbZ87lzUZZeXqhRdnrfpJOznLIjht bmAqhslLZu9qTDN2kO+6WAE0OjrvqWEwn8VnxIA94gm84zbMm5FaEunTGotGB3ciXA6U id9r/EcpVlXoK1aHeSrKpy9aXQj6To+4mhXmYxgWPb7n9vgFEbDfhJ2/VGgXbAFYYYVd YGpOxH30yT9BcPkbiFvqLjqlQRyoieW84YAwGp06vZouSkKKiTNJMeasFsTxC8jGryFf Kd/Ya4E0Wx3aQIzJfTPh4CJjPx0a0E+cezcQLKUXK4AFRbKEPbOE42voNQzo0vpYjqHV IgIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=s8kGYtwqxuQjBjt50ABAKFfnnbLYJ+Tb8OFykkxbMfI=; b=Ni4NZt0ewC8ec1+fwKSIHkQyaUc/siHpWKgI8kFB/siw9U/ChRYhWLwG+mk9i3cf79 oFp7G5ObFOlQcKAszV8Cms+5Yx9kdSCPrxIEmHNEDPV7De9HJuXmhiDFnFyiBC4j5qgj EP2l6lIIomFdB550eKPwQz3otbpim7Q0ae0WdMHOpolmXp1q39Xy7qMwqcRYEEg1gKtB cuU1Yv6YMfefl/LLeXIGFGCwQ/TMa6Q2lNzlnc0RD/qTPEOYuvka50OfQJA+dAJf1KG7 5Wkza4sxGe5EFxzBVgqQxfcK9+mtwAZfyv4QC9A3396MdHHt1uUUIhfLEi0DsH+HwjGi i78g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id h48-v6si8848199ede.163.2018.06.18.02.18.28 for (version=TLS1 cipher=AES128-SHA bits=128/128); Mon, 18 Jun 2018 02:18:28 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (charybdis-ext-too.suse.de [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id CD9E6ABEA; Mon, 18 Jun 2018 09:18:27 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , linux-api@vger.kernel.org, Christoph Lameter , David Rientjes , Mel Gorman , Matthew Wilcox , Vlastimil Babka Subject: [PATCH v2 2/7] mm, slab/slub: introduce kmalloc-reclaimable caches Date: Mon, 18 Jun 2018 11:18:03 +0200 Message-Id: <20180618091808.4419-3-vbabka@suse.cz> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180618091808.4419-1-vbabka@suse.cz> References: <20180618091808.4419-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Kmem caches can be created with a SLAB_RECLAIM_ACCOUNT flag, which indicates they contain objects which can be reclaimed under memory pressure (typically through a shrinker). This makes the slab pages accounted as NR_SLAB_RECLAIMABLE in vmstat, which is reflected also the MemAvailable meminfo counter and in overcommit decisions. The slab pages are also allocated with __GFP_RECLAIMABLE, which is good for anti-fragmentation through grouping pages by mobility. The generic kmalloc-X caches are created without this flag, but sometimes are used also for objects that can be reclaimed, which due to varying size cannot have a dedicated kmem cache with SLAB_RECLAIM_ACCOUNT flag. A prominent example are dcache external names, which prompted the creation of a new, manually managed vmstat counter NR_INDIRECTLY_RECLAIMABLE_BYTES in commit f1782c9bc547 ("dcache: account external names as indirectly reclaimable memory"). To better handle this and any other similar cases, this patch introduces SLAB_RECLAIM_ACCOUNT variants of kmalloc caches, named kmalloc-rcl-X. They are used whenever the kmalloc() call passes __GFP_RECLAIMABLE among gfp flags. They are added to the kmalloc_caches array as a new type. Allocations with both __GFP_DMA and __GFP_RECLAIMABLE will use a dma type cache. This change only applies to SLAB and SLUB, not SLOB. This is fine, since SLOB's target are tiny system and this patch does add some overhead of kmem management objects. Signed-off-by: Vlastimil Babka --- include/linux/slab.h | 16 +++++++++++---- mm/slab_common.c | 48 ++++++++++++++++++++++++++++---------------- 2 files changed, 43 insertions(+), 21 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 4299c59353a1..d89e934e0d8b 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -296,11 +296,12 @@ static inline void __check_heap_object(const void *ptr, unsigned long n, (KMALLOC_MIN_SIZE) : 16) #define KMALLOC_NORMAL 0 +#define KMALLOC_RECLAIM 1 #ifdef CONFIG_ZONE_DMA -#define KMALLOC_DMA 1 -#define KMALLOC_TYPES 2 +#define KMALLOC_DMA 2 +#define KMALLOC_TYPES 3 #else -#define KMALLOC_TYPES 1 +#define KMALLOC_TYPES 2 #endif #ifndef CONFIG_SLOB @@ -309,12 +310,19 @@ extern struct kmem_cache *kmalloc_caches[KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; static __always_inline unsigned int kmalloc_type(gfp_t flags) { int is_dma = 0; + int is_reclaimable; #ifdef CONFIG_ZONE_DMA is_dma = !!(flags & __GFP_DMA); #endif - return is_dma; + is_reclaimable = !!(flags & __GFP_RECLAIMABLE); + + /* + * If an allocation is botth __GFP_DMA and __GFP_RECLAIMABLE, return + * KMALLOC_DMA and effectively ignore __GFP_RECLAIMABLE + */ + return (is_dma * 2) + (is_reclaimable & !is_dma); } /* diff --git a/mm/slab_common.c b/mm/slab_common.c index 635f2d8d0198..8a30d6979936 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1103,10 +1103,21 @@ void __init setup_kmalloc_cache_index_table(void) } } -static void __init new_kmalloc_cache(int idx, slab_flags_t flags) +static void __init +new_kmalloc_cache(int idx, int type, slab_flags_t flags) { - kmalloc_caches[KMALLOC_NORMAL][idx] = create_kmalloc_cache( - kmalloc_info[idx].name, + const char *name; + + if (type == KMALLOC_RECLAIM) { + flags |= SLAB_RECLAIM_ACCOUNT; + name = kasprintf(GFP_NOWAIT, "kmalloc-rcl-%u", + kmalloc_info[idx].size); + BUG_ON(!name); + } else { + name = kmalloc_info[idx].name; + } + + kmalloc_caches[type][idx] = create_kmalloc_cache(name, kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); } @@ -1118,22 +1129,25 @@ static void __init new_kmalloc_cache(int idx, slab_flags_t flags) */ void __init create_kmalloc_caches(slab_flags_t flags) { - int i; - int type = KMALLOC_NORMAL; + int i, type; - for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { - if (!kmalloc_caches[type][i]) - new_kmalloc_cache(i, flags); + for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { + for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { + if (!kmalloc_caches[type][i]) + new_kmalloc_cache(i, type, flags); - /* - * Caches that are not of the two-to-the-power-of size. - * These have to be created immediately after the - * earlier power of two caches - */ - if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[type][1] && i == 6) - new_kmalloc_cache(1, flags); - if (KMALLOC_MIN_SIZE <= 64 && !kmalloc_caches[type][2] && i == 7) - new_kmalloc_cache(2, flags); + /* + * Caches that are not of the two-to-the-power-of size. + * These have to be created immediately after the + * earlier power of two caches + */ + if (KMALLOC_MIN_SIZE <= 32 && i == 6 && + !kmalloc_caches[type][1]) + new_kmalloc_cache(1, type, flags); + if (KMALLOC_MIN_SIZE <= 64 && i == 7 && + !kmalloc_caches[type][2]) + new_kmalloc_cache(2, type, flags); + } } /* Kmalloc array is now usable */