From patchwork Tue Jul 31 09:06:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vlastimil Babka X-Patchwork-Id: 10550489 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 68CF8139A for ; Tue, 31 Jul 2018 09:07:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5674F29FB4 for ; Tue, 31 Jul 2018 09:07:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4A8682A206; Tue, 31 Jul 2018 09:07:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A82FB29FB4 for ; Tue, 31 Jul 2018 09:07:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC4196B0266; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 92CC36B000C; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 77EB66B026A; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f72.google.com (mail-pl0-f72.google.com [209.85.160.72]) by kanga.kvack.org (Postfix) with ESMTP id 239796B0010 for ; Tue, 31 Jul 2018 05:07:03 -0400 (EDT) Received: by mail-pl0-f72.google.com with SMTP id az8-v6so10944511plb.15 for ; Tue, 31 Jul 2018 02:07:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=VZYOiAD3wrtrXYcfZKeldGvXRLaSYRD8JPqMQHNBua4=; b=iJOVbx+2fWqXTLx2Q2P6ZYsipCTko2yHPFPEFeW0a8lyS6c1dYX/UOWOk8JmdV31mn EPA2cyE7eb632c/x9DDUssH2m8Ge3Ga+JxF6sh2S/TZcwnm93rhdsO5ozThFPfYRjY3P 2ASeODXduCIDvJiSrDwWulQ12gM0bdCz1SEQl/9Jk4gMCQKudFDDustnTFqzc74RryQ3 PVSDFUjUohfhf9t/TdjGGmrlynfPFmaFb+MCqbJFCLH1sWq89Bj+Ccip+QahyAAHsso+ noLyyALFh3nyBA2Vq/AVAXfUV0nBq9o+qmpCT9zx9gz8G7syScp5Nm1Wz2BicLBQs65X lvrQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Gm-Message-State: AOUpUlFMz5xBwBqPJZny8DamP7K0t61ARV0GvIbADimOdVq/FfLj43wQ tfve7vYoNdyaT8rwGOcqjDr2vRfFL79gMm5rQYKk4BP2lTxRmRuLh0HDLTbHD2Y677QzOyPScJi NfvXTr4zU7VXkFK6Lkl4+j+kkzc2XhIRktF0il1FhJD8YNVeVq3Ut2Hewvu5SP4oJcQ== X-Received: by 2002:a62:2459:: with SMTP id r86-v6mr21242963pfj.31.1533028022807; Tue, 31 Jul 2018 02:07:02 -0700 (PDT) X-Google-Smtp-Source: AAOMgpcOn9aG4t4kaBnKBM9RbfPP7Q0tGA0wFMGXRQdRhIsDftSEH67uALHFZS1OgsbO2qDgIvLy X-Received: by 2002:a62:2459:: with SMTP id r86-v6mr21242914pfj.31.1533028021758; Tue, 31 Jul 2018 02:07:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1533028021; cv=none; d=google.com; s=arc-20160816; b=s5YCaaWfVlHJnvfZLG1OBBHg1cHOI+bkQlOUHkAe7iYvDlanJUFfU0QFDBe7xW7zWo ce5NhiDI6KjprSurT2WPXAs4z5172NQgRyg2sWQvls9EidLhFBC0+strzUU1VKmWZSVk zjYaQkox2P9wcxkYO+UCxtbo75gFoqXCktd5GKW5h3TUnckZTHHDWcdwFNaa/Z3ZWz69 UCtY8xAt8XrxR6ZtqoAX9DXUL3JhrR43dZAoB3D1kt3g+KBNGy+LLAA+OVM/G3as++4Z fZgscmfLsC+oxvOGi/d6897r/1+663P1Gi/nRZt2nYZ+LkSagbOtA3g+YyFNBopcGDfS 4eSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=VZYOiAD3wrtrXYcfZKeldGvXRLaSYRD8JPqMQHNBua4=; b=WsJBq2M4c4TPsRBI1c4tK6gBP3ou8/OB2RNWnpw8pckPj0RdvidYt7WhrO35DWfok+ QEDO5cnIms1oqN8mCzOKm6r1a/6UOAhwiW9dqLch55YcwvTfn/BoQMT2K8ICIPxyxSH1 mn2ZBVmwQsJYPNvp787IxCuN8qWgeAQixJAcUIFNL5nw0Rve1Pjy53TyTwGsAWIQ0FK8 LUubBgRQ6CpssLNg4VuBUJpE4BxIoITCwkEADE/BpEeHo2VZ4lsH6uwZP2RKPdpkvbvJ mSOhu0dEc/ZZX8f+Gz4b/p3Kyb8aT8g5N7n5HOrLPNNVY/lQKbSmf66XDA3GDVTQrkUy /oww== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz Received: from mx1.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id i21-v6si13452255pgg.513.2018.07.31.02.07.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 31 Jul 2018 02:07:01 -0700 (PDT) Received-SPF: pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) client-ip=195.135.220.15; Authentication-Results: mx.google.com; spf=pass (google.com: domain of vbabka@suse.cz designates 195.135.220.15 as permitted sender) smtp.mailfrom=vbabka@suse.cz X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay1.suse.de (unknown [195.135.220.254]) by mx1.suse.de (Postfix) with ESMTP id 8CF10AE1E; Tue, 31 Jul 2018 09:06:58 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, linux-api@vger.kernel.org, Roman Gushchin , Michal Hocko , Johannes Weiner , Christoph Lameter , David Rientjes , Joonsoo Kim , Mel Gorman , Matthew Wilcox , Vlastimil Babka Subject: [PATCH v4 2/6] mm, slab/slub: introduce kmalloc-reclaimable caches Date: Tue, 31 Jul 2018 11:06:45 +0200 Message-Id: <20180731090649.16028-3-vbabka@suse.cz> X-Mailer: git-send-email 2.18.0 In-Reply-To: <20180731090649.16028-1-vbabka@suse.cz> References: <20180731090649.16028-1-vbabka@suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Kmem caches can be created with a SLAB_RECLAIM_ACCOUNT flag, which indicates they contain objects which can be reclaimed under memory pressure (typically through a shrinker). This makes the slab pages accounted as NR_SLAB_RECLAIMABLE in vmstat, which is reflected also the MemAvailable meminfo counter and in overcommit decisions. The slab pages are also allocated with __GFP_RECLAIMABLE, which is good for anti-fragmentation through grouping pages by mobility. The generic kmalloc-X caches are created without this flag, but sometimes are used also for objects that can be reclaimed, which due to varying size cannot have a dedicated kmem cache with SLAB_RECLAIM_ACCOUNT flag. A prominent example are dcache external names, which prompted the creation of a new, manually managed vmstat counter NR_INDIRECTLY_RECLAIMABLE_BYTES in commit f1782c9bc547 ("dcache: account external names as indirectly reclaimable memory"). To better handle this and any other similar cases, this patch introduces SLAB_RECLAIM_ACCOUNT variants of kmalloc caches, named kmalloc-rcl-X. They are used whenever the kmalloc() call passes __GFP_RECLAIMABLE among gfp flags. They are added to the kmalloc_caches array as a new type. Allocations with both __GFP_DMA and __GFP_RECLAIMABLE will use a dma type cache. This change only applies to SLAB and SLUB, not SLOB. This is fine, since SLOB's target are tiny system and this patch does add some overhead of kmem management objects. Signed-off-by: Vlastimil Babka Acked-by: Mel Gorman Acked-by: Christoph Lameter Acked-by: Roman Gushchin --- include/linux/slab.h | 16 ++++++++++++++- mm/slab_common.c | 48 ++++++++++++++++++++++++++++---------------- 2 files changed, 46 insertions(+), 18 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index f35f2c1f37b9..b79a7cd59875 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -295,8 +295,13 @@ static inline void __check_heap_object(const void *ptr, unsigned long n, #define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \ (KMALLOC_MIN_SIZE) : 16) +/* + * Whenever changing this, take care of that kmalloc_type() and + * create_kmalloc_caches() still work as intended. + */ enum kmalloc_cache_type { KMALLOC_NORMAL = 0, + KMALLOC_RECLAIM, #ifdef CONFIG_ZONE_DMA KMALLOC_DMA, #endif @@ -310,12 +315,21 @@ kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH + 1]; static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) { int is_dma = 0; + int type_dma = 0; + int is_reclaimable; #ifdef CONFIG_ZONE_DMA is_dma = !!(flags & __GFP_DMA); + type_dma = is_dma * KMALLOC_DMA; #endif - return is_dma; + is_reclaimable = !!(flags & __GFP_RECLAIMABLE); + + /* + * If an allocation is both __GFP_DMA and __GFP_RECLAIMABLE, return + * KMALLOC_DMA and effectively ignore __GFP_RECLAIMABLE + */ + return type_dma + (is_reclaimable & !is_dma) * KMALLOC_RECLAIM; } /* diff --git a/mm/slab_common.c b/mm/slab_common.c index 4572941440f3..03f40b273ea3 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -1108,10 +1108,21 @@ void __init setup_kmalloc_cache_index_table(void) } } -static void __init new_kmalloc_cache(int idx, slab_flags_t flags) +static void __init +new_kmalloc_cache(int idx, int type, slab_flags_t flags) { - kmalloc_caches[KMALLOC_NORMAL][idx] = create_kmalloc_cache( - kmalloc_info[idx].name, + const char *name; + + if (type == KMALLOC_RECLAIM) { + flags |= SLAB_RECLAIM_ACCOUNT; + name = kasprintf(GFP_NOWAIT, "kmalloc-rcl-%u", + kmalloc_info[idx].size); + BUG_ON(!name); + } else { + name = kmalloc_info[idx].name; + } + + kmalloc_caches[type][idx] = create_kmalloc_cache(name, kmalloc_info[idx].size, flags, 0, kmalloc_info[idx].size); } @@ -1123,22 +1134,25 @@ static void __init new_kmalloc_cache(int idx, slab_flags_t flags) */ void __init create_kmalloc_caches(slab_flags_t flags) { - int i; - int type = KMALLOC_NORMAL; + int i, type; - for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { - if (!kmalloc_caches[type][i]) - new_kmalloc_cache(i, flags); + for (type = KMALLOC_NORMAL; type <= KMALLOC_RECLAIM; type++) { + for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) { + if (!kmalloc_caches[type][i]) + new_kmalloc_cache(i, type, flags); - /* - * Caches that are not of the two-to-the-power-of size. - * These have to be created immediately after the - * earlier power of two caches - */ - if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[type][1] && i == 6) - new_kmalloc_cache(1, flags); - if (KMALLOC_MIN_SIZE <= 64 && !kmalloc_caches[type][2] && i == 7) - new_kmalloc_cache(2, flags); + /* + * Caches that are not of the two-to-the-power-of size. + * These have to be created immediately after the + * earlier power of two caches + */ + if (KMALLOC_MIN_SIZE <= 32 && i == 6 && + !kmalloc_caches[type][1]) + new_kmalloc_cache(1, type, flags); + if (KMALLOC_MIN_SIZE <= 64 && i == 7 && + !kmalloc_caches[type][2]) + new_kmalloc_cache(2, type, flags); + } } /* Kmalloc array is now usable */