From patchwork Mon Jun 3 04:26:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972193 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 793CC14DB for ; Mon, 3 Jun 2019 04:27:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 68C2428174 for ; Mon, 3 Jun 2019 04:27:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5C08828387; Mon, 3 Jun 2019 04:27:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 881C228174 for ; Mon, 3 Jun 2019 04:27:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8EC276B026C; Mon, 3 Jun 2019 00:27:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 89B8E6B026D; Mon, 3 Jun 2019 00:27:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 78B276B026E; Mon, 3 Jun 2019 00:27:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id 5AA616B026C for ; Mon, 3 Jun 2019 00:27:43 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id c4so13709281qkd.16 for ; Sun, 02 Jun 2019 21:27:43 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=TNd3AKi13tITwVLf9iauaAEdzLmpPhHyUVYcM4rMbN8=; b=GM4gwCcjwlZxKE6l4EqWHMMyQ1skhzwypx4MZUvj7UjWnTyfwYCyPt5TASGmOT87X0 F/aERkNmZUKWahhryO6zmWXHvKNpEpr5gk72vObqHvtmIqwKvbOoIifUzoFsingYXTDr n+MiEXcjy0pTCqQ2f+iC9pXGgzJ6bx/UBWx1QsQ8nyO+hTHZYtOmvPxu9IhhcAScAGVF pAsXUyH18I39o50VkEYkKGIKPAd2CMIBxAlyD4j855Fke4dSHrMPDiN81gAIqERwKOdI pU0Gbn/Pqk37UdWqbGysbUxbNIBidiU1m6AVGK3pchVcKdJtMseuqxB58Q+zHKtJ49g6 JMWg== X-Gm-Message-State: APjAAAWxoSkfZAz31UjSP2rKFscZ9CpqkUmS2XNegS+i7P5FppLQQ6gr vNziV2kv6/Q7rYInHNXnPcTExTfwC+8eGGUa3DIW4JVOtw53Dyugf5DM8RNU77/0BRuTb6d6hRi cOwenTd7due5J8pkdxE2QDLVuVHYTiJjxEjR4f2lqlSihYk9zMlqy+Su+prByq2Y= X-Received: by 2002:a0c:d40d:: with SMTP id t13mr1167175qvh.175.1559536063089; Sun, 02 Jun 2019 21:27:43 -0700 (PDT) X-Google-Smtp-Source: APXvYqykKA3OzAgo0VsIYl+KVugKS/Fis3BSzVj54MbbiycoYylqTUxRgfGfUFFQFUJhVeV6Hgzx X-Received: by 2002:a0c:d40d:: with SMTP id t13mr1167106qvh.175.1559536061616; Sun, 02 Jun 2019 21:27:41 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536061; cv=none; d=google.com; s=arc-20160816; b=McxHp627GI4kPBAMwaPQdDMrfaRjY/H1cNYkwH3bI22RtmV/nW9FTD/MflvmwEuIfZ aXBosEWgAWrNdwY50toorqQf+XqMH2MsZWqqgZwuZU7WhZoHpzmEDqMBqInxD1YjxgFX RKHcajlFbncP2Qvf3i/Jdhb14thOtU0YaaFh2ncyrgqCKpssJVD0cSSrOumNQbGtxmZc mgG5fGLVHlBwjpbaITp+TDqAt0/3lfPKyhdRq5lIWx7gINyvUUfFbYiiUvVapXp7zG8+ oPeo+BQSW6KBAcJOhEsfKiF0Sg+vOuQGsYdxWyOfDAztB0JFeshI9N3OzHcfHJUq5P7d NjNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=TNd3AKi13tITwVLf9iauaAEdzLmpPhHyUVYcM4rMbN8=; b=fbJoh4QOMO1Im39c0S0Bac+bpkxt498OcNVRECLr/R5MwthRIk/jx0Ny4HGW4i4mtd AeTCMKRI4S8wQmzn6A718VvKLAPC410tXAD/GKlmg305GWNZNdBrQxskOPmovUzf21JY 0Ue68yp+uP2kqeJfMLpk3Q4+WStypE74R36wlGUDZrhZnQvfcijnadYjvLdjDe5CNON9 WAhwsbJ562W7cuGgODR98eCrfmW+Tq1Mq3OamqMEFc1iwOJ6dSqRHBbVJMd/2uzCvSEA MLZhhoaoeHym2q9CBod2sVexYZAxVTWKo6v3uy4o/evtl5ck9ZFydpff66+nfXWGMBtK HX/w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=Vkzc4LYp; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id s51si3151076qtk.50.2019.06.02.21.27.41 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:27:41 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=Vkzc4LYp; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 5DBEC1384; Mon, 3 Jun 2019 00:27:41 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:27:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=TNd3AKi13tITwVLf9iauaAEdzLmpPhHyUVYcM4rMbN8=; b=Vkzc4LYp /5xDnTNN+Nfu0iVRlIFqTfQOgB99QvwsXfrfDniMZ9IivTm59qe9tzXVm/U0Q1Y7 x7532FpV/ROI0VLbaDG0QtoevdcHoH392l23r7jC/XRm6tZhSpoRgaupLHljRTv9 w2qPO5xy3eFaHe7Lt/zBwuI718BNxvwYSUH1A+XSFL7UTJ6P6ktoPubdnCV+dY1z C+10JpvdYVlDyjMXs8JRBte4wT4N3jYIRtatRUrr9yQnk0HJWntBFZ6LEzSdm5nl 6CNarloB8shWAgZEy/bfhVPYeoHIMG/7npTKdrwQwTfKr+me+xwQi+37rBZavSxQ MkjIRNnPGzyW5w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id 3FBAA80061; Mon, 3 Jun 2019 00:27:34 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 01/15] slub: Add isolate() and migrate() methods Date: Mon, 3 Jun 2019 14:26:23 +1000 Message-Id: <20190603042637.2018-2-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the two methods needed for moving objects and enable the display of the callbacks via the /sys/kernel/slab interface. Add documentation explaining the use of these methods and the prototypes for slab.h. Add functions to setup the callbacks method for a slab cache. Add empty functions for SLAB/SLOB. The API is generic so it could be theoretically implemented for these allocators as well. Change sysfs 'ctor' field to be 'ops' to contain all the callback operations defined for a slab cache. Display the existing 'ctor' callback in the ops fields contents along with 'isolate' and 'migrate' callbacks. Signed-off-by: Tobin C. Harding --- include/linux/slab.h | 70 ++++++++++++++++++++++++++++++++++++++++ include/linux/slub_def.h | 3 ++ mm/slub.c | 59 +++++++++++++++++++++++++++++---- 3 files changed, 126 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..886fc130334d 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -154,6 +154,76 @@ void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); void memcg_destroy_kmem_caches(struct mem_cgroup *); +/* + * Function prototypes passed to kmem_cache_setup_mobility() to enable + * mobile objects and targeted reclaim in slab caches. + */ + +/** + * typedef kmem_cache_isolate_func - Object migration callback function. + * @s: The cache we are working on. + * @ptr: Pointer to an array of pointers to the objects to isolate. + * @nr: Number of objects in @ptr array. + * + * The purpose of kmem_cache_isolate_func() is to pin each object so that + * they cannot be freed until kmem_cache_migrate_func() has processed + * them. This may be accomplished by increasing the refcount or setting + * a flag. + * + * The object pointer array passed is also passed to + * kmem_cache_migrate_func(). The function may remove objects from the + * array by setting pointers to %NULL. This is useful if we can + * determine that an object is being freed because + * kmem_cache_isolate_func() was called when the subsystem was calling + * kmem_cache_free(). In that case it is not necessary to increase the + * refcount or specially mark the object because the release of the slab + * lock will lead to the immediate freeing of the object. + * + * Context: Called with locks held so that the slab objects cannot be + * freed. We are in an atomic context and no slab operations + * may be performed. + * Return: A pointer that is passed to the migrate function. If any + * objects cannot be touched at this point then the pointer may + * indicate a failure and then the migration function can simply + * remove the references that were already obtained. The private + * data could be used to track the objects that were already pinned. + */ +typedef void *kmem_cache_isolate_func(struct kmem_cache *s, void **ptr, int nr); + +/** + * typedef kmem_cache_migrate_func - Object migration callback function. + * @s: The cache we are working on. + * @ptr: Pointer to an array of pointers to the objects to migrate. + * @nr: Number of objects in @ptr array. + * @node: The NUMA node where the object should be allocated. + * @private: The pointer returned by kmem_cache_isolate_func(). + * + * This function is responsible for migrating objects. Typically, for + * each object in the input array you will want to allocate an new + * object, copy the original object, update any pointers, and free the + * old object. + * + * After this function returns all pointers to the old object should now + * point to the new object. + * + * Context: Called with no locks held and interrupts enabled. Sleeping + * is possible. Any operation may be performed. + */ +typedef void kmem_cache_migrate_func(struct kmem_cache *s, void **ptr, + int nr, int node, void *private); + +/* + * kmem_cache_setup_mobility() is used to setup callbacks for a slab cache. + */ +#ifdef CONFIG_SLUB +void kmem_cache_setup_mobility(struct kmem_cache *, kmem_cache_isolate_func, + kmem_cache_migrate_func); +#else +static inline void +kmem_cache_setup_mobility(struct kmem_cache *s, kmem_cache_isolate_func isolate, + kmem_cache_migrate_func migrate) {} +#endif + /* * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above. diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index d2153789bd9f..2879a2f5f8eb 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -99,6 +99,9 @@ struct kmem_cache { gfp_t allocflags; /* gfp flags to use on each alloc */ int refcount; /* Refcount for slab cache destroy */ void (*ctor)(void *); + kmem_cache_isolate_func *isolate; + kmem_cache_migrate_func *migrate; + unsigned int inuse; /* Offset to metadata */ unsigned int align; /* Alignment */ unsigned int red_left_pad; /* Left redzone padding size */ diff --git a/mm/slub.c b/mm/slub.c index cd04dbd2b5d0..1c380a2bc78a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4317,6 +4317,33 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return err; } +void kmem_cache_setup_mobility(struct kmem_cache *s, + kmem_cache_isolate_func isolate, + kmem_cache_migrate_func migrate) +{ + /* + * Mobile objects must have a ctor otherwise the object may be + * in an undefined state on allocation. Since the object may + * need to be inspected by the migration function at any time + * after allocation we must ensure that the object always has a + * defined state. + */ + if (!s->ctor) { + pr_err("%s: require constructor to setup mobility\n", s->name); + return; + } + + s->isolate = isolate; + s->migrate = migrate; + + /* + * Sadly serialization requirements currently mean that we have + * to disable fast cmpxchg based processing. + */ + s->flags &= ~__CMPXCHG_DOUBLE; +} +EXPORT_SYMBOL(kmem_cache_setup_mobility); + void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller) { struct kmem_cache *s; @@ -5001,13 +5028,33 @@ static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf, } SLAB_ATTR(cpu_partial); -static ssize_t ctor_show(struct kmem_cache *s, char *buf) +static int op_show(char *buf, const char *txt, unsigned long addr) { - if (!s->ctor) - return 0; - return sprintf(buf, "%pS\n", s->ctor); + int x = 0; + + x += sprintf(buf, "%s : ", txt); + x += sprint_symbol(buf + x, addr); + x += sprintf(buf + x, "\n"); + + return x; +} + +static ssize_t ops_show(struct kmem_cache *s, char *buf) +{ + int x = 0; + + if (s->ctor) + x += op_show(buf + x, "ctor", (unsigned long)s->ctor); + + if (s->isolate) + x += op_show(buf + x, "isolate", (unsigned long)s->isolate); + + if (s->migrate) + x += op_show(buf + x, "migrate", (unsigned long)s->migrate); + + return x; } -SLAB_ATTR_RO(ctor); +SLAB_ATTR_RO(ops); static ssize_t aliases_show(struct kmem_cache *s, char *buf) { @@ -5420,7 +5467,7 @@ static struct attribute *slab_attrs[] = { &objects_partial_attr.attr, &partial_attr.attr, &cpu_slabs_attr.attr, - &ctor_attr.attr, + &ops_attr.attr, &aliases_attr.attr, &align_attr.attr, &hwcache_align_attr.attr, From patchwork Mon Jun 3 04:26:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972197 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 975861515 for ; Mon, 3 Jun 2019 04:27:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 864CA28174 for ; Mon, 3 Jun 2019 04:27:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 795CB28387; Mon, 3 Jun 2019 04:27:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C997428174 for ; Mon, 3 Jun 2019 04:27:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BD2BF6B026D; Mon, 3 Jun 2019 00:27:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BA9B96B026E; Mon, 3 Jun 2019 00:27:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A726E6B026F; Mon, 3 Jun 2019 00:27:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id 833DA6B026D for ; Mon, 3 Jun 2019 00:27:50 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id w184so13732407qka.15 for ; Sun, 02 Jun 2019 21:27:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=TF57F9Io+O0/OZ6gKFCnEeu96OOBr/TUAaoumLTsEF0=; b=e0pyZiKQfuHsWJx6oh/F7DKSnCcJcbfjJkhyh+K1RyHBz7ws/BL0aK4c0GEvptK82C EzBsZoWZ6aJdSQdC26S7eEISlFX9kWW7LRLyGY2swBFLlgQw3BdiMI1AfGkqX2WFN+Qr k270lA7FQgEeqpapgE4qWp+wdPIVGnMlOVLbI2HiZYG11RXec5pIzYpEy3Nw/G8eArYX 2GbPAJYk6okl7WzmUXBoecmcEDfG0M32Ypngd56FcjrEeS59NnsJsz49bQ6kZcRj8jLY 5wCNq7c6VxeBqq4QlAqD4cKHyd0ZR1PJQQ+AKDY31DnU19SjzvLGd/HARwhZgl0djCbo kXNw== X-Gm-Message-State: APjAAAV9YBs61dT6UGE4VTolL/8Sat9RkFo/Nqc0b/B9IiydDsF0q/xb 7KODHW3S0Iu8U8/QEpP3SagJG8OZXVhlD+YkiEEbXMWKQc7UmPc6fNdDmKE17RKIjQLeV/GI1HJ DVnw1Cc0btQGKRRpQQxilLd123t55OfNt8nfpOsEKQYigIUszPeJ3dL3YDa0PrqY= X-Received: by 2002:ad4:45ab:: with SMTP id y11mr20366475qvu.137.1559536070268; Sun, 02 Jun 2019 21:27:50 -0700 (PDT) X-Google-Smtp-Source: APXvYqyFZgTXhyrKpB6CPef14JLunfDiDEjEWAy5xxVyHiHV9vk/7ACtjycFxs9NcCruLCTM1tg4 X-Received: by 2002:ad4:45ab:: with SMTP id y11mr20366425qvu.137.1559536068956; Sun, 02 Jun 2019 21:27:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536068; cv=none; d=google.com; s=arc-20160816; b=We+vr7gARBoQyd7FojlkZWTfwvJu975fuhQK8QOVs4fdiAcqyzA7MBw2tl4ECTrYS6 d/B3QPiWmUqVeKSORLBn52HWPLeJeA7jQmnjt7/8kCYYvk+esSEsISX/qP22vDTuu+fW 93MOkqYaj/kC5tEd65izLLDdRSoR/4YB9vShzA5AgtmJO1OI5ppGAKnAITlLAYKbuVxV L5uaugVj3O2N5aXHhd+bAAgNXM1z891zpbM3JSVdDiY5xSWgptmzn5Mz77N7+vPcaqeL DlwtmgY3jgJmwgv4UtjfWhIYxnaFwBaMegIAYEaiFbirBTq1aF0G4YqOnJcnL67htRrx BdBw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=TF57F9Io+O0/OZ6gKFCnEeu96OOBr/TUAaoumLTsEF0=; b=l7eRV4fF+o4h2/S04ve+XIHmbksol8qPIQfF6Vjz4x9JKl17U3CaMakCLl6Dms7R/R boQAduLZUPtOGrMNNm2PldyaMPE0rg10tipEdyzmi5jYoakdAQDkTGI247DHSbnUJL9I odzIRnibHsJ6V9JjUl7ALdziUbrxqwtzwO3nJxDjaW7DBh+daWF+TrOAReiUeJbJUBEc QztqvYZ2Zw5vQ+QOrPziaMQNcr5U3eZBaJUzi1PKTflN/50q12rJXoJsUvd+6JVJZLsT ftgmd95NLOhd5bOH2HVc3yZFSFdVlkgrXBlj8T7uxf4L0dhak4gSlR78aKHpr2Z6txux wZew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="X/3bMxR6"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id 53si6281013qtx.195.2019.06.02.21.27.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:27:48 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="X/3bMxR6"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id B4C2A123E; Mon, 3 Jun 2019 00:27:48 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:27:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=TF57F9Io+O0/OZ6gKFCnEeu96OOBr/TUAaoumLTsEF0=; b=X/3bMxR6 mB9zUnWZJzuM+Bprvs6pFmCIdoAyFvitCZBf/23ycwHUxne94eTRA+JPme/UrzN3 XwciTBJUhW/wqk/QV/gdfq5jeav2AAMN1z3Q33gNKcCu7zSHpavA/ePKKGkxh+hd y3udydsyz+QrzS7ED8ZRu0q5uTNg3uB1XS+4OB65EetjPoSMC+MHqP+F0zZYMCAd o1xYRFrCM8xSeCJy8FLiWzbfit/TGx5HOE5TNeri2lACHTmXCr2P/MzBpi89M7KI iVlWWLR/pbU0NSBhnqVseiCGvtj3B1YbkJWUZqxwboo4ysHeEky8wXIuiXO4Vf9R 1ZGXTPsqj1K5pQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id 7931F8005B; Mon, 3 Jun 2019 00:27:41 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 02/15] tools/vm/slabinfo: Add support for -C and -M options Date: Mon, 3 Jun 2019 14:26:24 +1000 Message-Id: <20190603042637.2018-3-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP -C lists caches that use a ctor. -M lists caches that support object migration. Add command line options to show caches with a constructor and caches that are movable (i.e. have migrate function). Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 40 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index 73818f1b2ef8..cbfc56c44c2f 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -33,6 +33,7 @@ struct slabinfo { unsigned int hwcache_align, object_size, objs_per_slab; unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; + int movable, ctor; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; unsigned long free_fastpath, free_slowpath; @@ -67,6 +68,8 @@ int show_report; int show_alias; int show_slab; int skip_zero = 1; +int show_movable; +int show_ctor; int show_numa; int show_track; int show_first_alias; @@ -109,11 +112,13 @@ static void fatal(const char *x, ...) static void usage(void) { - printf("slabinfo 4/15/2011. (c) 2007 sgi/(c) 2011 Linux Foundation.\n\n" - "slabinfo [-aADefhilnosrStTvz1LXBU] [N=K] [-dafzput] [slab-regexp]\n" + printf("slabinfo 4/15/2017. (c) 2007 sgi/(c) 2011 Linux Foundation/(c) 2017 Jump Trading LLC.\n\n" + "slabinfo [-aACDefhilMnosrStTvz1LXBU] [N=K] [-dafzput] [slab-regexp]\n" + "-a|--aliases Show aliases\n" "-A|--activity Most active slabs first\n" "-B|--Bytes Show size in bytes\n" + "-C|--ctor Show slabs with ctors\n" "-D|--display-active Switch line format to activity\n" "-e|--empty Show empty slabs\n" "-f|--first-alias Show first alias\n" @@ -121,6 +126,7 @@ static void usage(void) "-i|--inverted Inverted list\n" "-l|--slabs Show slabs\n" "-L|--Loss Sort by loss\n" + "-M|--movable Show caches that support movable objects\n" "-n|--numa Show NUMA information\n" "-N|--lines=K Show the first K slabs\n" "-o|--ops Show kmem_cache_ops\n" @@ -588,6 +594,12 @@ static void slabcache(struct slabinfo *s) if (show_empty && s->slabs) return; + if (show_ctor && !s->ctor) + return; + + if (show_movable && !s->movable) + return; + if (sort_loss == 0) store_size(size_str, slab_size(s)); else @@ -602,6 +614,10 @@ static void slabcache(struct slabinfo *s) *p++ = '*'; if (s->cache_dma) *p++ = 'd'; + if (s->ctor) + *p++ = 'C'; + if (s->movable) + *p++ = 'M'; if (s->hwcache_align) *p++ = 'A'; if (s->poison) @@ -636,7 +652,8 @@ static void slabcache(struct slabinfo *s) printf("%-21s %8ld %7d %15s %14s %4d %1d %3ld %3ld %s\n", s->name, s->objects, s->object_size, size_str, dist_str, s->objs_per_slab, s->order, - s->slabs ? (s->partial * 100) / s->slabs : 100, + s->slabs ? (s->partial * 100) / + (s->slabs * s->objs_per_slab) : 100, s->slabs ? (s->objects * s->object_size * 100) / (s->slabs * (page_size << s->order)) : 100, flags); @@ -1256,6 +1273,13 @@ static void read_slab_dir(void) slab->alloc_node_mismatch = get_obj("alloc_node_mismatch"); slab->deactivate_bypass = get_obj("deactivate_bypass"); chdir(".."); + if (read_slab_obj(slab, "ops")) { + if (strstr(buffer, "ctor :")) + slab->ctor = 1; + if (strstr(buffer, "migrate :")) + slab->movable = 1; + } + if (slab->name[0] == ':') alias_targets++; slab++; @@ -1332,6 +1356,8 @@ static void xtotals(void) } struct option opts[] = { + { "ctor", no_argument, NULL, 'C' }, + { "movable", no_argument, NULL, 'M' }, { "aliases", no_argument, NULL, 'a' }, { "activity", no_argument, NULL, 'A' }, { "debug", optional_argument, NULL, 'd' }, @@ -1367,7 +1393,7 @@ int main(int argc, char *argv[]) page_size = getpagesize(); - while ((c = getopt_long(argc, argv, "aAd::Defhil1noprstvzTSN:LXBU", + while ((c = getopt_long(argc, argv, "aACd::Defhil1MnoprstvzTSN:LXBU", opts, NULL)) != -1) switch (c) { case '1': @@ -1376,6 +1402,9 @@ int main(int argc, char *argv[]) case 'a': show_alias = 1; break; + case 'C': + show_ctor = 1; + break; case 'A': sort_active = 1; break; @@ -1399,6 +1428,9 @@ int main(int argc, char *argv[]) case 'i': show_inverted = 1; break; + case 'M': + show_movable = 1; + break; case 'n': show_numa = 1; break; From patchwork Mon Jun 3 04:26:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972201 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 33E1F14DB for ; Mon, 3 Jun 2019 04:27:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 244A428174 for ; Mon, 3 Jun 2019 04:27:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1438F28387; Mon, 3 Jun 2019 04:27:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A330F28174 for ; Mon, 3 Jun 2019 04:27:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 800F86B026E; Mon, 3 Jun 2019 00:27:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7B1CF6B026F; Mon, 3 Jun 2019 00:27:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A0126B0270; Mon, 3 Jun 2019 00:27:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 491ED6B026E for ; Mon, 3 Jun 2019 00:27:57 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id v4so13725216qkj.10 for ; Sun, 02 Jun 2019 21:27:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=jGU9saE69a/gfeKWv7WhcnJU8nruwl7/fOnxwG1r6J8=; b=dBN4XmfhbeZrW9pekC1s+97mEqQUEOrMF27mWCFcX+v2z9eETiemx6gp2kbKAE1gPW ihlLM6hQ4+2HVJh2AMjvRAGc+uDnhOoWgug43rYEo8cCQk4qwO6DsUmFnRjDHK2UhGhm eU6zNHaFWRk0zNz6ejoSr4gc40uHKz44rYe7b+zNmS8gY/yjg2pQiiaSLzPvYRjvHX1K aImOQTFaax59DZCSK1pxLZhl6SzDYgZhneru5433wFQAqmdWvbJpFmZFYse/lkLG97F+ E2OGYrzvr9mDt/Hh9YmurjMByjSvTFq5uIbIcOMcye2kn4SYKP4KL3P+Wr55ihTdZN9+ 9EdA== X-Gm-Message-State: APjAAAWBcVaQE8x4xubfb9bRFhN2NW9x3dtHFA9hXhiK95gzUWgyhVAX CilNMLvQZ7ZdujpdsZ6iX9akRpd2k+H3bPQZV0MvoPyF4iNfZXeB3Y/0wzvbQDiASA5vXrW5eDG ZhJ+wHZvJ0sALJXj7xyNVrv3uBCSbZJeWwSZYGSa2It+wqHO8XY53SIrSY3qKY9o= X-Received: by 2002:ac8:374b:: with SMTP id p11mr20363240qtb.316.1559536077038; Sun, 02 Jun 2019 21:27:57 -0700 (PDT) X-Google-Smtp-Source: APXvYqzHjx+Zqnu61NnZcpGoXSqZgFM7tNaCAEV0w8xDJ1p1vzfhZd6NUKTItWkgCFowHWnqDhlY X-Received: by 2002:ac8:374b:: with SMTP id p11mr20363211qtb.316.1559536076183; Sun, 02 Jun 2019 21:27:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536076; cv=none; d=google.com; s=arc-20160816; b=Xcvy58p40uUobexFLUecMdgMIt3M2S/MHUtxTFhoAS5M+TDAnG+/nhlD8FwZj7944J mVlkDisVhvCyZ7R3HuiCN56s8HZSk/S0Lf0egEYNAOp11pRIUiuyG0j0a44K0fCpi7UO 2w+VqRZcOBK74hfMUqwhANtlmSBhSDM0hR3xFXGLSliBfJcdIBNQ9pqoZ6Kb2iP80SO4 ZI8ojYWtnQnjX0NzlcDRp0EG5b/ztJP4VgVge3XMRjLkIkgg3tZOUt1ZgXo/30q00Hg0 zSddNqbwDFTY5R7vu6Z5Kv//InRB2T/ocf/eJvXiA15IwQTk6dkoAc8Eq/9Wo1BK1rop vMSw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=jGU9saE69a/gfeKWv7WhcnJU8nruwl7/fOnxwG1r6J8=; b=vhXkYHvQWx5kMzm0TX7fn4o8wcaVfuXlzrkdY950z0c1Mi9/TFZeg1JBviY5XLma2f wIqhrGgpMMY39ouaA+/pxdBiRmodUU6kaedGbjb+Q9ME7cycWQWht9U1q9zY6D/UUu38 99GDBUUw9JFMclBGXHWVepXqU0htQcvMgbDCSDUD6A4c90wOBcBRiY4bnmjVeDvu3yJ4 1H5A0zpBySEC3Sn9HDTYaIpp7wbWxdBbm0v3NVfIycDCoi3syV0E9bWT/dac3xkv2IWM H5Sla+s9hjWvXw3LeJ179Z3SBe2eXcKyCIvq3OwEd4+hq+yqKx/krOwUcmBdhjuKsLq7 eoqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=8AcGMewM; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id u35si1709638qvg.156.2019.06.02.21.27.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:27:56 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=8AcGMewM; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id E7B57123E; Mon, 3 Jun 2019 00:27:55 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:27:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=jGU9saE69a/gfeKWv7WhcnJU8nruwl7/fOnxwG1r6J8=; b=8AcGMewM Ndtg4pdRtsPZpOOKqHUApLCaiiXKUPEyBBAaPdJaaMoZ4+B5eR9/1K70X18Cqxcv UsCHyzJfIMF1H80xNpYpRnilppJPkHuSiN7XKRXkAkLJxGDpxJ5+AxWk/on2Kj1Q 4Dgq5Inh5H9ZWVDSXbCwgyxV1G5i+CbiPvRiY6x/sXaOvU84KptkfhMgi2NnGMi4 7I5PYAJ7yq34ekuIxckmPeilZtbhnrt1WlolEFtEiRgzu/StEyM3hs4sJ0RA0fdX oT2kh8XjDLPSLm/nvFzdE9m/Fj/VLX+MNaxFJHKRQiG3KrYdgq8oL73l0Ab+WN57 LZoMA1Af0XMAwQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id BBC6680059; Mon, 3 Jun 2019 00:27:48 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 03/15] slub: Sort slab cache list Date: Mon, 3 Jun 2019 14:26:25 +1000 Message-Id: <20190603042637.2018-4-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It is advantageous to have all defragmentable slabs together at the beginning of the list of slabs so that there is no need to scan the complete list. Put defragmentable caches first when adding a slab cache and others last. Signed-off-by: Tobin C. Harding --- mm/slab_common.c | 2 +- mm/slub.c | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..db5e9a0b1535 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -393,7 +393,7 @@ static struct kmem_cache *create_cache(const char *name, goto out_free_cache; s->refcount = 1; - list_add(&s->list, &slab_caches); + list_add_tail(&s->list, &slab_caches); memcg_link_cache(s); out: if (err) diff --git a/mm/slub.c b/mm/slub.c index 1c380a2bc78a..66d474397c0f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4333,6 +4333,8 @@ void kmem_cache_setup_mobility(struct kmem_cache *s, return; } + mutex_lock(&slab_mutex); + s->isolate = isolate; s->migrate = migrate; @@ -4341,6 +4343,10 @@ void kmem_cache_setup_mobility(struct kmem_cache *s, * to disable fast cmpxchg based processing. */ s->flags &= ~__CMPXCHG_DOUBLE; + + list_move(&s->list, &slab_caches); /* Move to top */ + + mutex_unlock(&slab_mutex); } EXPORT_SYMBOL(kmem_cache_setup_mobility); From patchwork Mon Jun 3 04:26:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972205 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C85931515 for ; Mon, 3 Jun 2019 04:28:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B71F928174 for ; Mon, 3 Jun 2019 04:28:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AA39B28387; Mon, 3 Jun 2019 04:28:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC4DE28174 for ; Mon, 3 Jun 2019 04:28:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C47306B026F; Mon, 3 Jun 2019 00:28:05 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BF7446B0270; Mon, 3 Jun 2019 00:28:05 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE6A66B0271; Mon, 3 Jun 2019 00:28:05 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 87A5B6B026F for ; Mon, 3 Jun 2019 00:28:05 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id n126so13676591qkc.18 for ; Sun, 02 Jun 2019 21:28:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=XMu/tqjBTMx3ouEMQ23wmZhVjiUN3IGgccdmUpS11IQ=; b=uLgj8UEF9QtPajwVHBO+zoTWEN9e22sLNuNAKp5drxJb9blyuW6FT1vh9NSokCuCWG quDs8AwvAq2PALnioPv4PsN/t3s8pMHxtQPj/n6sMfd9IcJohNClMLtBY95tXSqIJrVW LsCew1OqLCqHMOsH8cfgKMlsSIxXRDP+1KZ9+EeHwCGrrlxkKV4c94tRFhknlSAkPawO MirhQdD04gu1bTvMZhiXyG1TAgTyYTwtFwey+ihyswIp1hje+qP1nb0YQMsuQBv7a4RM eWFubDfjKt6ke9PLgqGcNjVomnlPMUW/z3rSb8Q71FGAnjlzPpIxMGORxTJGLAmHYMeQ gppw== X-Gm-Message-State: APjAAAWtSK75v3XrV6tM5UOrARuR7sHHoprPYX7VoNob37fnM24fe0tY /QehlUZX+xO+WiwtqAoPXGzCXjdCSoWZ+fWQhpf4uL03Nk4xdGlJfCcNTMg/HBkuza2xL1vu7Ss l0rxYVS7IflLxT21UAd6wQhF0VNA7XD19jxH0mod2axJutSMMG7EdOl5XS8uXCVk= X-Received: by 2002:a0c:ad23:: with SMTP id u32mr1055361qvc.39.1559536085231; Sun, 02 Jun 2019 21:28:05 -0700 (PDT) X-Google-Smtp-Source: APXvYqzhcOpQgm3DJcPk1v1fgd0KzVRY25VMmuMxYDyHFW72v4nXv4AXpcTYWo+JHURPGgagHMsc X-Received: by 2002:a0c:ad23:: with SMTP id u32mr1055304qvc.39.1559536083562; Sun, 02 Jun 2019 21:28:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536083; cv=none; d=google.com; s=arc-20160816; b=jpJtrRsmtE8I0b+oP58SsrbXE0PICboVshNV3lHu8+tzJI0TD/GjkbomkdOg6knF0i A8uFwBFS5B2HvwjhPVNa0HQ43G4WdvlGD8dH4sHR938EwDt1IKuxeyWCMYhKrMN+0DRs iDJ/ICEsbyPEhtfYp88joSUPjNR+NLiubLKivtTd7BIHApTd6vXBQo8d2TkGhXSxLqyM h3qLJln7254Q28C6tlU8UNSZ3dbPzzO704zRMNHBY5D6Ge9T2rCeCHks9EHq5OmHwp2h 4t4Z/h5oZ5uLYEVe8VdsViAHupyVfo60B2gy1nYqzgwv0UhfdHgmlLhCd/FSHJhl4rq/ abtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=XMu/tqjBTMx3ouEMQ23wmZhVjiUN3IGgccdmUpS11IQ=; b=eOYAyhGD+BAFkNV5IKUKdIfvWc+x1t/2EsM+a3b27B9ZYsdfcaFJCre8m0xXHSqfQ7 +RhaRwleguABDPjxq5deFrCjhS6gHP6CZvhGgBA7dJ4iiGEk2RxIKfWJskiZjOGyX+JK ACILruT3mcRtp4c7cHY8/oDFtusdThedZQUxrO+h5TWZ7SJKgSVtEET8ym7vzdfnha7b QXNwyirl7FXyRtJZT1DwRW73y3ZxfydB34OAZo7xj+phbPIsaHvdtRi8VqLNNNcAW4/y lXShh3AvKhuszY1PMDhIrURychD6LlKFfjCgbG9M39hVgu7X1UGGoMvYPk1097o8BY9B D6pQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=bjawKjQ6; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id 39si1273199qtr.30.2019.06.02.21.28.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:28:03 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=bjawKjQ6; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 524431912; Mon, 3 Jun 2019 00:28:03 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:28:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=XMu/tqjBTMx3ouEMQ23wmZhVjiUN3IGgccdmUpS11IQ=; b=bjawKjQ6 tRuPf4xwrMmqmyCDKP36NkmUgSz9GHpHrr1F3LwsD8v6AuFTwrLaIFoYlVAFY97w iYJFNd5ygb2J8ZDTY3eh8WepGXQEHGKPNQzdo7WKE6xzDY7UEWX2j2o8rECIAk4H JwgWXVPjlBy+iVX3VGkXtXwaYsMj5hZpCZmjQdWLNecOcN9NpL4fxM3rNvyYNqPp 1Ot8nbolRRVc9aXf6W0acgvVEY1sdvO8Bwq4UqZFZADhyajs9Dm8dgwK0K2kZuLi rzH1n7jJrAPEv4yruzEpfS3P7iaV/KRHzBgU0eCKeoDlKoVhP0Q9Z6CB0E88SXYm jiFIIv4icSN2RA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id F053B8005C; Mon, 3 Jun 2019 00:27:55 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 04/15] slub: Slab defrag core Date: Mon, 3 Jun 2019 14:26:26 +1000 Message-Id: <20190603042637.2018-5-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Internal fragmentation can occur within pages used by the slub allocator. Under some workloads large numbers of pages can be used by partial slab pages. This under-utilisation is bad simply because it wastes memory but also because if the system is under memory pressure higher order allocations may become difficult to satisfy. If we can defrag slab caches we can alleviate these problems. Implement Slab Movable Objects in order to defragment slab caches. Slab defragmentation may occur: 1. Unconditionally when __kmem_cache_shrink() is called on a slab cache by the kernel calling kmem_cache_shrink(). 2. Unconditionally through the use of the slabinfo command. slabinfo -s 3. Conditionally via the use of kmem_cache_defrag() - Use Slab Movable Objects when shrinking cache. Currently when the kernel calls kmem_cache_shrink() we curate the partial slabs list. If object migration is not enabled for the cache we still do this, if however, SMO is enabled we attempt to move objects in partially full slabs in order to defragment the cache. Shrink attempts to move all objects in order to reduce the cache to a single partial slab for each node. - Add conditional per node defrag via new function: kmem_defrag_slabs(int node). kmem_defrag_slabs() attempts to defragment all slab caches for node. Defragmentation is done conditionally dependent on MAX_PARTIAL _and_ defrag_used_ratio. Caches are only considered for defragmentation if the number of partial slabs exceeds MAX_PARTIAL (per node). Also, defragmentation only occurs if the usage ratio of the slab is lower than the configured percentage (sysfs field added in this patch). Fragmentation ratios are measured by calculating the percentage of objects in use compared to the total number of objects that the slab page can accommodate. The scanning of slab caches is optimized because the defragmentable slabs come first on the list. Thus we can terminate scans on the first slab encountered that does not support defragmentation. kmem_defrag_slabs() takes a node parameter. This can either be -1 if defragmentation should be performed on all nodes, or a node number. Defragmentation may be disabled by setting defrag ratio to 0 echo 0 > /sys/kernel/slab//defrag_used_ratio - Add a defrag ratio sysfs field and set it to 30% by default. A limit of 30% specifies that more than 3 out of 10 available slots for objects need to be in use otherwise slab defragmentation will be attempted on the remaining objects. In order for a cache to be defragmentable the cache must support object migration (SMO). Enabling SMO for a cache is done via a call to the recently added function: void kmem_cache_setup_mobility(struct kmem_cache *, kmem_cache_isolate_func, kmem_cache_migrate_func); Signed-off-by: Tobin C. Harding --- Documentation/ABI/testing/sysfs-kernel-slab | 14 + include/linux/slab.h | 1 + include/linux/slub_def.h | 7 + mm/slub.c | 385 ++++++++++++++++---- 4 files changed, 334 insertions(+), 73 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-kernel-slab b/Documentation/ABI/testing/sysfs-kernel-slab index 29601d93a1c2..8bd893968e4f 100644 --- a/Documentation/ABI/testing/sysfs-kernel-slab +++ b/Documentation/ABI/testing/sysfs-kernel-slab @@ -180,6 +180,20 @@ Description: list. It can be written to clear the current count. Available when CONFIG_SLUB_STATS is enabled. +What: /sys/kernel/slab/cache/defrag_used_ratio +Date: June 2019 +KernelVersion: 5.2 +Contact: Christoph Lameter + Pekka Enberg , +Description: + The defrag_used_ratio file allows the control of how aggressive + slab fragmentation reduction works at reclaiming objects from + sparsely populated slabs. This is a percentage. If a slab has + less than this percentage of objects allocated then reclaim will + attempt to reclaim objects so that the whole slab page can be + freed. 0% specifies no reclaim attempt (defrag disabled), 100% + specifies attempt to reclaim all pages. The default is 30%. + What: /sys/kernel/slab/cache/deactivate_to_tail Date: February 2008 KernelVersion: 2.6.25 diff --git a/include/linux/slab.h b/include/linux/slab.h index 886fc130334d..4bf381b34829 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -149,6 +149,7 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, void (*ctor)(void *)); void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); +unsigned long kmem_defrag_slabs(int node); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index 2879a2f5f8eb..34c6f1250652 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -107,6 +107,13 @@ struct kmem_cache { unsigned int red_left_pad; /* Left redzone padding size */ const char *name; /* Name (only for display!) */ struct list_head list; /* List of slab caches */ + int defrag_used_ratio; /* + * Ratio used to check against the + * percentage of objects allocated in a + * slab page. If less than this ratio + * is allocated then reclaim attempts + * are made. + */ #ifdef CONFIG_SYSFS struct kobject kobj; /* For sysfs */ struct work_struct kobj_remove_work; diff --git a/mm/slub.c b/mm/slub.c index 66d474397c0f..2157205df7ba 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -355,6 +355,12 @@ static __always_inline void slab_lock(struct page *page) bit_spin_lock(PG_locked, &page->flags); } +static __always_inline int slab_trylock(struct page *page) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + return bit_spin_trylock(PG_locked, &page->flags); +} + static __always_inline void slab_unlock(struct page *page) { VM_BUG_ON_PAGE(PageTail(page), page); @@ -3634,6 +3640,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) set_cpu_partial(s); + s->defrag_used_ratio = 30; #ifdef CONFIG_NUMA s->remote_node_defrag_ratio = 1000; #endif @@ -3950,79 +3957,6 @@ void kfree(const void *x) } EXPORT_SYMBOL(kfree); -#define SHRINK_PROMOTE_MAX 32 - -/* - * kmem_cache_shrink discards empty slabs and promotes the slabs filled - * up most to the head of the partial lists. New allocations will then - * fill those up and thus they can be removed from the partial lists. - * - * The slabs with the least items are placed last. This results in them - * being allocated from last increasing the chance that the last objects - * are freed in them. - */ -int __kmem_cache_shrink(struct kmem_cache *s) -{ - int node; - int i; - struct kmem_cache_node *n; - struct page *page; - struct page *t; - struct list_head discard; - struct list_head promote[SHRINK_PROMOTE_MAX]; - unsigned long flags; - int ret = 0; - - flush_all(s); - for_each_kmem_cache_node(s, node, n) { - INIT_LIST_HEAD(&discard); - for (i = 0; i < SHRINK_PROMOTE_MAX; i++) - INIT_LIST_HEAD(promote + i); - - spin_lock_irqsave(&n->list_lock, flags); - - /* - * Build lists of slabs to discard or promote. - * - * Note that concurrent frees may occur while we hold the - * list_lock. page->inuse here is the upper limit. - */ - list_for_each_entry_safe(page, t, &n->partial, slab_list) { - int free = page->objects - page->inuse; - - /* Do not reread page->inuse */ - barrier(); - - /* We do not keep full slabs on the list */ - BUG_ON(free <= 0); - - if (free == page->objects) { - list_move(&page->slab_list, &discard); - n->nr_partial--; - } else if (free <= SHRINK_PROMOTE_MAX) - list_move(&page->slab_list, promote + free - 1); - } - - /* - * Promote the slabs filled up most to the head of the - * partial list. - */ - for (i = SHRINK_PROMOTE_MAX - 1; i >= 0; i--) - list_splice(promote + i, &n->partial); - - spin_unlock_irqrestore(&n->list_lock, flags); - - /* Release empty slabs */ - list_for_each_entry_safe(page, t, &discard, slab_list) - discard_slab(s, page); - - if (slabs_node(s, node)) - ret = 1; - } - - return ret; -} - #ifdef CONFIG_MEMCG static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) { @@ -4317,6 +4251,287 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return err; } +/* + * Allocate a slab scratch space that is sufficient to keep pointers to + * individual objects for all objects in cache and also a bitmap for the + * objects (used to mark which objects are active). + */ +static inline void *alloc_scratch(struct kmem_cache *s) +{ + unsigned int size = oo_objects(s->max); + + return kmalloc(size * sizeof(void *) + + BITS_TO_LONGS(size) * sizeof(unsigned long), + GFP_KERNEL); +} + +/* + * move_slab_page() - Move all objects in the given slab. + * @page: The slab we are working on. + * @scratch: Pointer to scratch space. + * @node: The target node to move objects to. + * + * If the target node is not the current node then the object is moved + * to the target node. If the target node is the current node then this + * is an effective way of defragmentation since the current slab page + * with its object is exempt from allocation. + */ +static void move_slab_page(struct page *page, void *scratch, int node) +{ + unsigned long objects; + struct kmem_cache *s; + unsigned long flags; + unsigned long *map; + void *private; + int count; + void *p; + void **vector = scratch; + void *addr = page_address(page); + + local_irq_save(flags); + slab_lock(page); + + BUG_ON(!PageSlab(page)); /* Must be a slab page */ + BUG_ON(!page->frozen); /* Slab must have been frozen earlier */ + + s = page->slab_cache; + objects = page->objects; + map = scratch + objects * sizeof(void **); + + /* Determine used objects */ + bitmap_fill(map, objects); + for (p = page->freelist; p; p = get_freepointer(s, p)) + __clear_bit(slab_index(p, s, addr), map); + + /* Build vector of pointers to objects */ + count = 0; + memset(vector, 0, objects * sizeof(void **)); + for_each_object(p, s, addr, objects) + if (test_bit(slab_index(p, s, addr), map)) + vector[count++] = p; + + if (s->isolate) + private = s->isolate(s, vector, count); + else + /* Objects do not need to be isolated */ + private = NULL; + + /* + * Pinned the objects. Now we can drop the slab lock. The slab + * is frozen so it cannot vanish from under us nor will + * allocations be performed on the slab. However, unlocking the + * slab will allow concurrent slab_frees to proceed. So the + * subsystem must have a way to tell from the content of the + * object that it was freed. + * + * If neither RCU nor ctor is being used then the object may be + * modified by the allocator after being freed which may disrupt + * the ability of the migrate function to tell if the object is + * free or not. + */ + slab_unlock(page); + local_irq_restore(flags); + + /* Perform callback to move the objects */ + s->migrate(s, vector, count, node, private); +} + +/* + * kmem_cache_defrag() - Defragment node. + * @s: cache we are working on. + * @node: The node to move objects from. + * @target_node: The node to move objects to. + * @ratio: The defrag ratio (percentage, between 0 and 100). + * + * Release slabs with zero objects and try to call the migration function + * for slabs with less than the 'ratio' percentage of objects allocated. + * + * Moved objects are allocated on @target_node. + * + * Return: The number of partial slabs left on @node after the + * operation. + */ +static unsigned long kmem_cache_defrag(struct kmem_cache *s, + int node, int target_node, int ratio) +{ + struct kmem_cache_node *n = get_node(s, node); + struct page *page, *page2; + LIST_HEAD(move_list); + unsigned long flags; + + if (node == target_node && n->nr_partial <= 1) { + /* + * Trying to reduce fragmentation on a node but there is + * only a single or no partial slab page. This is already + * the optimal object density that we can reach. + */ + return n->nr_partial; + } + + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &n->partial, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + if (page->inuse) { + if (page->inuse > ratio * page->objects / 100) { + slab_unlock(page); + /* + * Skip slab because the object density + * in the slab page is high enough. + */ + continue; + } + + list_move(&page->lru, &move_list); + if (s->migrate) { + /* Stop page being considered for allocations */ + n->nr_partial--; + page->frozen = 1; + } + slab_unlock(page); + } else { /* Empty slab page */ + list_del(&page->lru); + n->nr_partial--; + slab_unlock(page); + discard_slab(s, page); + } + } + + if (!s->migrate) { + /* + * No defrag method. By simply putting the zaplist at + * the end of the partial list we can let them simmer + * longer and thus increase the chance of all objects + * being reclaimed. + */ + list_splice(&move_list, n->partial.prev); + } + + spin_unlock_irqrestore(&n->list_lock, flags); + + if (s->migrate && !list_empty(&move_list)) { + void **scratch = alloc_scratch(s); + if (scratch) { + /* Try to remove / move the objects left */ + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + } + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + /* + * Objects left in slab page, move it to the + * tail of the partial list to increase the + * chance that the freeing of the remaining + * objects will free the slab page. + */ + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); + } + + return n->nr_partial; +} + +/** + * kmem_defrag_slabs() - Defrag slab caches. + * @node: The node to defrag or -1 for all nodes. + * + * Defrag slabs conditional on the amount of fragmentation in a page. + * + * Return: The total number of partial slabs in migratable caches left + * on @node after the operation. + */ +unsigned long kmem_defrag_slabs(int node) +{ + struct kmem_cache *s; + unsigned long left = 0; + int nid; + + if (node >= MAX_NUMNODES) + return -EINVAL; + + /* + * kmem_defrag_slabs() may be called from the reclaim path which + * may be called for any page allocator alloc. So there is the + * danger that we get called in a situation where slub already + * acquired the slub_lock for other purposes. + */ + if (!mutex_trylock(&slab_mutex)) + return 0; + + list_for_each_entry(s, &slab_caches, list) { + /* + * Defragmentable caches come first. If the slab cache is + * not defragmentable then we can stop traversing the list. + */ + if (!s->migrate) + break; + + if (node >= 0) { + if (s->node[node]->nr_partial > MAX_PARTIAL) { + left += kmem_cache_defrag(s, node, node, + s->defrag_used_ratio); + } + continue; + } + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (s->node[nid]->nr_partial > MAX_PARTIAL) { + left += kmem_cache_defrag(s, nid, nid, + s->defrag_used_ratio); + } + } + } + mutex_unlock(&slab_mutex); + return left; +} +EXPORT_SYMBOL(kmem_defrag_slabs); + +/** + * __kmem_cache_shrink() - Shrink a cache. + * @s: The cache to shrink. + * + * Reduces the memory footprint of a slab cache by as much as possible. + * + * This works by: + * 1. Removing empty slabs from the partial list. + * 2. Migrating slab objects to denser slab pages if the slab cache + * supports migration. If not, reorganizing the partial list so that + * more densely allocated slab pages come first. + * + * Not called directly, called by kmem_cache_shrink(). + */ +int __kmem_cache_shrink(struct kmem_cache *s) +{ + int node; + int left = 0; + + flush_all(s); + for_each_node_state(node, N_NORMAL_MEMORY) + left += kmem_cache_defrag(s, node, node, 100); + + return left; +} +EXPORT_SYMBOL(__kmem_cache_shrink); + void kmem_cache_setup_mobility(struct kmem_cache *s, kmem_cache_isolate_func isolate, kmem_cache_migrate_func migrate) @@ -5168,6 +5383,29 @@ static ssize_t destroy_by_rcu_show(struct kmem_cache *s, char *buf) } SLAB_ATTR_RO(destroy_by_rcu); +static ssize_t defrag_used_ratio_show(struct kmem_cache *s, char *buf) +{ + return sprintf(buf, "%d\n", s->defrag_used_ratio); +} + +static ssize_t defrag_used_ratio_store(struct kmem_cache *s, + const char *buf, size_t length) +{ + unsigned long ratio; + int err; + + err = kstrtoul(buf, 10, &ratio); + if (err) + return err; + + if (ratio > 100) + return -EINVAL; + + s->defrag_used_ratio = ratio; + return length; +} +SLAB_ATTR(defrag_used_ratio); + #ifdef CONFIG_SLUB_DEBUG static ssize_t slabs_show(struct kmem_cache *s, char *buf) { @@ -5492,6 +5730,7 @@ static struct attribute *slab_attrs[] = { &validate_attr.attr, &alloc_calls_attr.attr, &free_calls_attr.attr, + &defrag_used_ratio_attr.attr, #endif #ifdef CONFIG_ZONE_DMA &cache_dma_attr.attr, From patchwork Mon Jun 3 04:26:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972209 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C67E314DB for ; Mon, 3 Jun 2019 04:28:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3BCF28174 for ; Mon, 3 Jun 2019 04:28:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A6FA328387; Mon, 3 Jun 2019 04:28:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37DCD28174 for ; Mon, 3 Jun 2019 04:28:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 234F66B0270; Mon, 3 Jun 2019 00:28:12 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1E50D6B0271; Mon, 3 Jun 2019 00:28:12 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D40C6B0272; Mon, 3 Jun 2019 00:28:12 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id E04676B0270 for ; Mon, 3 Jun 2019 00:28:11 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id v4so13725594qkj.10 for ; Sun, 02 Jun 2019 21:28:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=A2xUm/c1QMOPSqZ7s8invRNgw1dO9YvZCU++wFciDxknLwnnri447S0fAgd0UN/XzL iho+84Qerf3EebB5h6vumKF+jQ3ZQsaXrWXPigXrAP6pERpkzmaYpfyObl03ugehq2OY aD/A565xaRL2X5/pAn7OeNRQU0oPkeTTGVFJ1sxVjqGl6xlKTpIdH/DtO91AZ7kzWG9X bv6V0qUPTX3gikWHk2bIkrVqSPBeT5jlFY9sJ1d1qZjJ8My5tz4UUODtP7YehMqkpKn8 mN2ItEJWsO6MbEw2z4BzV+9blpAiXmJXdUapWuCTApWzJCHvg0uWoqo9vJuKUHN1UdAW UtDQ== X-Gm-Message-State: APjAAAVb+BEGFPD5eWJv6hzbaFcdBnLvezWYJQGZ2tsw8aqXeZRnD8sC IgTIupbmtoxFyCD5oZlzaq73l1hjDz4Suv4ZgMuONcx+F2xtfgD3ieG+yJmECz8KLkjbtHhXc5H ioQ3gTDqOgKYt3Ird/m+ZesrKjV1cJ4jzEYqmIEteBXo/cBHDo0GkYn1sp6mgOLc= X-Received: by 2002:ae9:f10b:: with SMTP id k11mr19526113qkg.238.1559536091694; Sun, 02 Jun 2019 21:28:11 -0700 (PDT) X-Google-Smtp-Source: APXvYqxroNCz1irGJpVmNwQfeuFXRY8pDddV5tDDi8e/g9CQX9ldLC3uGyPtGV0Dl39DtrRCA5fm X-Received: by 2002:ae9:f10b:: with SMTP id k11mr19526092qkg.238.1559536090899; Sun, 02 Jun 2019 21:28:10 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536090; cv=none; d=google.com; s=arc-20160816; b=zeClZUGEjTaogN2xhDLkJpDTPJePmJ42jIreIYbjwyyfQhBxTEQ0NVnjVZPEZc/PUd ghcxh0iS3+fTO5iNzHFl5hMgnRhZndSZdnnq4VlkXGe6h60xlrF1WI464x/QgnTyoauY fUkQ+8dYooyXCgXg5KpOFKSHWzWGVIRSyR8uQpiIPxDM5yObf9T1VWys16eFZVFNt6Nk 4YsXdMDiJeP1lPxUEJtk61L2EZCSp1km81yYJbGoEQVAMQlxICmtEfbzXI1+YgydKA0F oIgNzif7Gt3wBiZJCEhne517MK7ZGepANzglqBgFYIuLdSJ0kVPVdSeAXjY9PodbV0CK a3+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=W45ND3tRsH08+gQGzuJ0QbV997yAX+G1KX5gmjYCxhuOAHrF0/YCjBjm9qdHgEtEc5 nb3FzlsnNAn9+uJ8owqXlcPoEPaF12lw0islfBQ7DYlXAeL1thrjmrFkyBA4tyZatnic aEroIZUiIWN63K/n4r5cpSvDANooF/yjaspPlsrML0DnR27q1oS1QqmttrYX0ncZKnIB MGttdLMRUVjjWHz/dE+7c+ZicT30BDvO0VnD6xb5oSLgp6ArI2y9tIUa3VXE2GduK2tw n+m7R/LgLRG0H3haWD3TjcYhSIB/xT8XI8i+hyFekcdI36/G87cXBZ6ZLBiVt0Cy68ky KV5A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=W7fL1H8A; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id l13si4550405qtj.64.2019.06.02.21.28.10 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:28:10 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=W7fL1H8A; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id A5FB8199B; Mon, 3 Jun 2019 00:28:10 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:28:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=W7fL1H8A LfNFoRx+lIXkmmLAS7NFbIPAVxMml8jtELjMnYRAAQTRWlvVo6/ARYkxSziQ9YCC tulqD83nAZAExVG6cANgAuO4b5lFNSApeMfzM+5S4VJfghlVByJlKWLaI94eimpI LH19vEgir+ydq3UJ7NiGyeP2GErX9mh1pfFJ8psisUrzv0z0XiBW8my7Ys3PkcX4 rBt4QlzOZ6UPWo2JaCaYN8CIfADTxgTeJbE7LODwiRVF2U+PHCrFVWVvSKKYG2e3 3dPkek2wOblmPrRvAaTZvPHngGxyZhcT055s68QKl/s86utRsUbNt/GrJ2NRLEMq 9RJ1gH49Cy+RlQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id 70EDD80059; Mon, 3 Jun 2019 00:28:03 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 05/15] tools/vm/slabinfo: Add remote node defrag ratio output Date: Mon, 3 Jun 2019 14:26:27 +1000 Message-Id: <20190603042637.2018-6-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add output line for NUMA remote node defrag ratio. Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index cbfc56c44c2f..d2c22f9ee2d8 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -34,6 +34,7 @@ struct slabinfo { unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; int movable, ctor; + int remote_node_defrag_ratio; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; unsigned long free_fastpath, free_slowpath; @@ -377,6 +378,10 @@ static void slab_numa(struct slabinfo *s, int mode) if (skip_zero && !s->slabs) return; + if (mode) { + printf("\nNUMA remote node defrag ratio: %3d\n", + s->remote_node_defrag_ratio); + } if (!line) { printf("\n%-21s:", mode ? "NUMA nodes" : "Slab"); for(node = 0; node <= highest_node; node++) @@ -1272,6 +1277,8 @@ static void read_slab_dir(void) slab->cpu_partial_free = get_obj("cpu_partial_free"); slab->alloc_node_mismatch = get_obj("alloc_node_mismatch"); slab->deactivate_bypass = get_obj("deactivate_bypass"); + slab->remote_node_defrag_ratio = + get_obj("remote_node_defrag_ratio"); chdir(".."); if (read_slab_obj(slab, "ops")) { if (strstr(buffer, "ctor :")) From patchwork Mon Jun 3 04:26:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972213 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E87A61515 for ; Mon, 3 Jun 2019 04:28:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8D2228385 for ; Mon, 3 Jun 2019 04:28:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CC08E28174; Mon, 3 Jun 2019 04:28:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58DF628174 for ; Mon, 3 Jun 2019 04:28:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5DE416B026A; Mon, 3 Jun 2019 00:28:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5B7C36B0271; Mon, 3 Jun 2019 00:28:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A7936B0272; Mon, 3 Jun 2019 00:28:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 2CB586B026A for ; Mon, 3 Jun 2019 00:28:19 -0400 (EDT) Received: by mail-qt1-f199.google.com with SMTP id z16so6499112qto.10 for ; Sun, 02 Jun 2019 21:28:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=I0uSbvoSbvWkj+RnhubY9vN6KVomjpVhK9fTbQPZ6P5a9VGjeS+QJHcHqhZEst94NW fOwAhxpMHH9ltoNAzCnT+LRJVMxrcBDPqJO4qYxAPDKQ7TOqAAB0qMQKc252JFv8+Q30 VVLpoc8LRCSdxsrBnusR89wRyMxHP14qpL58HVSy/A1T8VF+9Jbl5iJu1zdzu7ordG6R cnpNMlm2zt11NRte1Lh6cz8DvtxDuvyN0L3ujW/kCrc17fdBull0mgiPvmg+2U6Fynt3 SQef+1RTeKmLUzvbAChu65RqXA1DBDsXXNftlX7kV6SE0i7KaqIz3z1qKjhhUwyME4Dn uBGQ== X-Gm-Message-State: APjAAAUsqAF4cORazEWF/cELNm1tgKKg+ZQZJDOgItVR8/QochyfBVnG P80Id1wIrq96NxeLpApryf2N12iKBVFuhLiGBhWPMkRJgezBPOyETxN0EjprlpWpvhMqO+KMhte 0XPq0ShVPxXam1Yy/xHKbVyxrzt6QYnBGo28CJzM22XVbOqqmWq7oApzP1UncePg= X-Received: by 2002:a0c:fcc6:: with SMTP id i6mr9543104qvq.109.1559536098939; Sun, 02 Jun 2019 21:28:18 -0700 (PDT) X-Google-Smtp-Source: APXvYqyOTpjhCS4n9ScPTSH+qieBgRjGKoj/37kNiPvf+DK+Xwh6elqGCHKl8iwG45AQi6pl8r/6 X-Received: by 2002:a0c:fcc6:: with SMTP id i6mr9543068qvq.109.1559536098093; Sun, 02 Jun 2019 21:28:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536098; cv=none; d=google.com; s=arc-20160816; b=KoXpRdmdOJFMuFQKl7YZi7OLLvpOm2SuECATSLNjBQRqK70SuRMJESY6wjZfDrhStW wdUTODQMiL9PNRC/AH31bZ7gIHxJ3sPZr3xK1q6MG2Bbk8DUCooEFEUdJaMilm0Pk4UI B3o7Cnbf8j24mhiocqO620D0zBUDPPkFrocyfsWEfudr4cZvWUte/ipvuUccgq22MJev 17CaPUXsSX2xHkaMo7gtw85Wl8tMqonpm9Di7TE6ad04G1VhuaXm/4gVR/sUv0K93eMz 8PB+2tQ4d/a6ctUwSHvbfqMN6bhujKqObZaefquCni5ygZoJTHgf8doy9/baFH/fXgCH MJ1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=O4inlAWb/f1pJY3jFYX6NVzWM3gWicRTJ7JeE9SXUKPFAWDPJScRernGcG2VlTOYsi Pbz2WNZnIi7yK7dMhhX4GdWrUOZbylLUIviHdRpEzYv2DsrCmnyhPpTS4uZV9hsshW5r W5CfqaAp6aBA9nNuL2j8xNmheSAHuXIa8fthIZY6Dnjf0NbthTNbA/5tbqiLOnlfgqg8 /AMfUUe3zuVyFcSmBo6Ojb9VcLyUyZQnDxzDZ6zzl5k3Hbp+ZkSD7qGyA9IT48MeCbik S7uqoKjSa/D/6gMEA4wy0enPTn5yiOAGp4N7dZKU112MNbMbUN4aVTbcxCc0EXhaYz8b brVg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=h8lhB1kH; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id q5si1009048qke.271.2019.06.02.21.28.18 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:28:18 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=h8lhB1kH; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id D64B415E4; Mon, 3 Jun 2019 00:28:17 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:28:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=h8lhB1kH /ZzWq7shkCDu3x1p3MQLgFcheSThypTFlbmAunpKVRrTI+jS3wEx0oXjx1+h4vae osoL6PSLNto2hLwr/Oqkoq9YUQ4/Ilv19G4cbmgdd/fe2oZ8QwkalEhSlW+fhfe0 78jdTuCqOqOjx5O1ekAIDMFSnqksJyvWPvmUfHSE1kYsHDlWzcY/CIGHeH7z3O/f 7f8EP/7M9IMXnrQQcoQ1jnISJkxFZKEp0KbxSVO5aU2648H3JH3VCqb5LP1aN3xY kytxgvMZjJY614JPtc0ry8yak8OPJjPoI4i+Kzdn/KwuUx/PPUx15JGQeBWK05ju LaW42Ofr9GVSGg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id AEE378005C; Mon, 3 Jun 2019 00:28:10 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 06/15] tools/vm/slabinfo: Add defrag_used_ratio output Date: Mon, 3 Jun 2019 14:26:28 +1000 Message-Id: <20190603042637.2018-7-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add output for the newly added defrag_used_ratio sysfs knob. Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index d2c22f9ee2d8..ef4ff93df4cc 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -34,6 +34,7 @@ struct slabinfo { unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; int movable, ctor; + int defrag_used_ratio; int remote_node_defrag_ratio; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; @@ -549,6 +550,8 @@ static void report(struct slabinfo *s) printf("** Slabs are destroyed via RCU\n"); if (s->reclaim_account) printf("** Reclaim accounting active\n"); + if (s->movable) + printf("** Defragmentation at %d%%\n", s->defrag_used_ratio); printf("\nSizes (bytes) Slabs Debug Memory\n"); printf("------------------------------------------------------------------------\n"); @@ -1279,6 +1282,7 @@ static void read_slab_dir(void) slab->deactivate_bypass = get_obj("deactivate_bypass"); slab->remote_node_defrag_ratio = get_obj("remote_node_defrag_ratio"); + slab->defrag_used_ratio = get_obj("defrag_used_ratio"); chdir(".."); if (read_slab_obj(slab, "ops")) { if (strstr(buffer, "ctor :")) From patchwork Mon Jun 3 04:26:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972217 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 50D1014DB for ; Mon, 3 Jun 2019 04:28:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E14128174 for ; Mon, 3 Jun 2019 04:28:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3021128387; Mon, 3 Jun 2019 04:28:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B064428174 for ; Mon, 3 Jun 2019 04:28:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8016A6B0271; Mon, 3 Jun 2019 00:28:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7D7F76B0272; Mon, 3 Jun 2019 00:28:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 679C66B0273; Mon, 3 Jun 2019 00:28:28 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 43B116B0271 for ; Mon, 3 Jun 2019 00:28:28 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id i64so6527045qtd.3 for ; Sun, 02 Jun 2019 21:28:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=fbpofKIVtxm0j2hLoYbHWx/4MF5IIfRTzKuE9DsjYInzaPWKpRM21uqQjo1aKzwdkz t1hSSA3iYOsUAx1v04I3fDeprOBK9mOVXuMGhXe6uEzxDY/r7IedmUxNsXXmehIRpP99 qIWvt/sr+Rukdz0hHD6uvpbjyMBjdjKVKMwYHN54J8mX3vxY8cUSoG7FctAogB6FQcJ1 XpvFezAChBSJsnq+1k5Pd8eWCEKNuhrrSYKwxaDxPdNGUumy3+zNxxrCrs4cIFRb7hZy VZR61mcCIWaz9t8uX/j+D7yn6JNc680V0KM++tyWlAybphq9+v8bxZWHoDXXKGyD8T7i 0c9A== X-Gm-Message-State: APjAAAX3SP3lwggcy41ITVCbZ7h8pABoOiDcOXoemVyoI37512MTOYMD uGgovs71neFk8jIFmSpliri5A3CumMIjzljDL5TaCFw5E62bKjoD4Sy6jOEcB46WmLfaw7RL5ec wEFCYv0n3mMUbw1YI+yfEXP1gMkrYJXl3ULVYtPdQv36J2aD7tYy9jUGyzZpL6ew= X-Received: by 2002:a37:b044:: with SMTP id z65mr19946155qke.294.1559536107874; Sun, 02 Jun 2019 21:28:27 -0700 (PDT) X-Google-Smtp-Source: APXvYqw+VmJhm47/78Kvl/3LKRbtwKj43mtEh1nfavpcn5Ty2BMXq1UJahMv90bBj05t+PzwsKie X-Received: by 2002:a37:b044:: with SMTP id z65mr19946083qke.294.1559536105789; Sun, 02 Jun 2019 21:28:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536105; cv=none; d=google.com; s=arc-20160816; b=oUbT9SsUmz7PDiPhglzOtomKMZyFZ7MQdlciqkXDRm3v0NDFuge23BRpyxKFWDhd1u n8JyDBSWcI23+AaPcga/T8AU6gdoZFIJGFrTcCuNk/FgvSF4B5gRS0nZrr4wAlcxfg64 Us9MmAn67M10EKfXAmfDb5ETgeA5Kc4OQ4lSwPTP1mhPaBA/IOW8545/0nldLFgEXwXQ hXXvnWAM9AnvSd8E7mwbVa4jDljcG+1DXN09t0JqtVOIyK/XSR7gK9fdGxzi9OcOShRY MI6y7LAvCwbY4PJPwXyeTN0DewmrYxwdNFyYlf9Fu/d+zn5czAIz/hLZ9/SOfVF3lSmO wvQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=ZIQTbsBzVg68eXvSqgzMd7V0wNryV052TH84tqfVKv7DiuJLJ8dZUcxmKS3fd4xq9z WH8neCueeVx7RyBO3C0ec2dKyGV1cGVhEU/B9nxoX2Jg6Nop5AbKM0/vEqzIKI/9SELG VlV6ViQOrBL1kSDTs3HSxtnM8SOmpbfeMD53r5UkPrrfezkRwZ9dLd4T6+XPhY28wH0g 5LVns48KzzandkcRF54Ypeq8XdygDm4j0IXGHh/Vx61Woylg/Enp7jRZIA8QI3gZn83q Z1C2dyGgFhb3cEuvQpwPAiG2YS80e/MlIJNhS5N9gr5T2Dcm6ShgBu7/DXNqXj72Q9Q1 YL0A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=otLA6iU7; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id x58si8822752qtx.298.2019.06.02.21.28.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:28:25 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=otLA6iU7; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 891BF1320; Mon, 3 Jun 2019 00:28:25 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:28:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=otLA6iU7 N5tKNA0bYh4PUMkA69Vsq+esgj0z7qDUSl5AM89g2a6ik2lW529QnCOAYRe5/ZWf WX9pqJ6eiVgpkwOIKA4HeWThZ6iBnJ8xZsJAE09MfSPMHebOYfkQIi84iXklyzDb jrj/1Xz4/mcumlE/mAC/EVbMHV40FVv7wdJCGf/nqiTLxOIDn9uDQJ+tIYGPqOFU btw1B6Yn8lSazDtLMo1x838KxoODbrxsWOouUwxbjT6AKvOyYpxgw93ZrU3tbt1z zpWiNK38lovUBZgagh6Ux13tSznYPTEmNt2M/fazMBRJ6AJsuHJq7MTu0PA06q7n I/udnjrra0mp3w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id EECB280059; Mon, 3 Jun 2019 00:28:17 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 07/15] tools/testing/slab: Add object migration test module Date: Mon, 3 Jun 2019 14:26:29 +1000 Message-Id: <20190603042637.2018-8-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just implemented slab movable objects for the SLUB allocator. We should test that code. In order to do so we need to be able to do a number of things - Create a cache - Enable Slab Movable Objects for the cache - Allocate objects to the cache - Free objects from within specific slabs of the cache We can do all this via a loadable module. Add a module that defines functions that can be triggered from userspace via a debugfs entry. From the source: /* * SLUB defragmentation a.k.a. Slab Movable Objects (SMO). * * This module is used for testing the SLUB allocator. Enables * userspace to run kernel functions via a debugfs file. * * debugfs: /sys/kernel/debugfs/smo/callfn (write only) * * String written to `callfn` is parsed by the module and associated * function is called. See fn_tab for mapping of strings to functions. */ References to allocated objects are kept by the module in a linked list so that userspace can control which object to free. We introduce the following four functions via the function table "enable": Enables object migration for the test cache. "alloc X": Allocates X objects "free X [Y]": Frees X objects starting at list position Y (default Y==0) "test": Runs [stress] tests from within the module (see below). {"enable", smo_enable_cache_mobility}, {"alloc", smo_alloc_objects}, {"free", smo_free_object}, {"test", smo_run_module_tests}, Freeing from the start of the list creates a hole in the slab being freed from (i.e. creates a partial slab). The results of running these commands can be see using `slabinfo` (available in tools/vm/): make -o slabinfo tools/vm/slabinfo.c Stress tests can be run from within the module. These tests are internal to the module because we verify that object references are still good after object migration. These are called 'stress' tests because it is intended that they create/free a lot of objects. Userspace can control the number of objects to create, default is 1000. Example test session -------------------- Relevant /proc/slabinfo column headers: name # mount -t debugfs none /sys/kernel/debug/ $ cd path/to/linux/tools/testing/slab; make ... # insmod slub_defrag.ko # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 0 0 392 20 2 From this we can see that the module created cache 'smo_test' with 20 objects per slab and 2 pages per slab (and cache is currently empty). We can play with the slab allocator manually: # insmod slub_defrag.ko # echo 'alloc 21' > callfn # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 21 40 392 20 2 We see here that 21 active objects have been allocated creating 2 slabs (40 total objects). # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 21 Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 2 Sanity Checks : On Total: 16384 SlabObj: 392 Full : 1 Redzoning : On Used : 1176 SlabSiz: 8192 Partial: 1 Poisoning : On Loss : 15208 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 7056 Align : 8 Objects: 20 Tracing : Off Lpadd: 704 Now free an object from the first slot of the first slab # echo 'free 1' > callfn # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 20 40 392 20 2 # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 20 Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 2 Sanity Checks : On Total: 16384 SlabObj: 392 Full : 0 Redzoning : On Used : 1120 SlabSiz: 8192 Partial: 2 Poisoning : On Loss : 15264 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 6720 Align : 8 Objects: 20 Tracing : Off Lpadd: 704 Calling shrink now on the cache does nothing because object migration is not enabled (output omitted). If we enable object migration then shrink the cache we expect the object from the second slab to me moved to the first slot in the first slab and the second slab to be removed from the partial list. # echo 'enable' > callfn # slabinfo smo_test --shrink # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 20 ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 1 Sanity Checks : On Total: 8192 SlabObj: 392 Full : 1 Redzoning : On Used : 1120 SlabSiz: 8192 Partial: 0 Poisoning : On Loss : 7072 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 6720 Align : 8 Objects: 20 Tracing : Off Lpadd: 352 We can run the stress tests (with the default number of objects): # cd /sys/kernel/debug/smo # echo 'test' > callfn [ 3.576617] smo: test using nr_objs: 1000 keep: 10 [ 3.580169] smo: Module tests completed successfully Signed-off-by: Tobin C. Harding --- tools/testing/slab/Makefile | 10 + tools/testing/slab/slub_defrag.c | 566 +++++++++++++++++++++++++++++++ 2 files changed, 576 insertions(+) create mode 100644 tools/testing/slab/Makefile create mode 100644 tools/testing/slab/slub_defrag.c diff --git a/tools/testing/slab/Makefile b/tools/testing/slab/Makefile new file mode 100644 index 000000000000..440c2e3e356f --- /dev/null +++ b/tools/testing/slab/Makefile @@ -0,0 +1,10 @@ +obj-m += slub_defrag.o + +KTREE=../../.. + +all: + make -C ${KTREE} M=$(PWD) modules + +clean: + make -C ${KTREE} M=$(PWD) clean + diff --git a/tools/testing/slab/slub_defrag.c b/tools/testing/slab/slub_defrag.c new file mode 100644 index 000000000000..4a5c24394b96 --- /dev/null +++ b/tools/testing/slab/slub_defrag.c @@ -0,0 +1,566 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * SLUB defragmentation a.k.a. Slab Movable Objects (SMO). + * + * This module is used for testing the SLUB allocator. Enables + * userspace to run kernel functions via a debugfs file. + * + * debugfs: /sys/kernel/debugfs/smo/callfn (write only) + * + * String written to `callfn` is parsed by the module and associated + * function is called. See fn_tab for mapping of strings to functions. + */ + +/* debugfs commands accept two optional arguments */ +#define SMO_CMD_DEFAUT_ARG -1 + +#define SMO_DEBUGFS_DIR "smo" +struct dentry *smo_debugfs_root; + +#define SMO_CACHE_NAME "smo_test" +static struct kmem_cache *cachep; + +struct smo_slub_object { + struct list_head list; + char buf[32]; /* Unused except to control size of object */ + long id; +}; + +/* Our list of allocated objects */ +LIST_HEAD(objects); + +static void list_add_to_objects(struct smo_slub_object *so) +{ + /* + * We free from the front of the list so store at the + * tail in order to put holes in the cache when we free. + */ + list_add_tail(&so->list, &objects); +} + +/** + * smo_object_ctor() - SMO object constructor function. + * @ptr: Pointer to memory where the object should be constructed. + */ +void smo_object_ctor(void *ptr) +{ + struct smo_slub_object *so = ptr; + + INIT_LIST_HEAD(&so->list); + memset(so->buf, 0, sizeof(so->buf)); + so->id = -1; +} + +/** + * smo_cache_migrate() - kmem_cache migrate function. + * @cp: kmem_cache pointer. + * @objs: Array of pointers to objects to migrate. + * @size: Number of objects in @objs. + * @node: NUMA node where the object should be allocated. + * @private: Pointer returned by kmem_cache_isolate_func(). + */ +void smo_cache_migrate(struct kmem_cache *cp, void **objs, int size, + int node, void *private) +{ + struct smo_slub_object **so_objs = (struct smo_slub_object **)objs; + struct smo_slub_object *so_old, *so_new; + int i; + + for (i = 0; i < size; i++) { + so_old = so_objs[i]; + + so_new = kmem_cache_alloc_node(cachep, GFP_KERNEL, node); + if (!so_new) { + pr_debug("kmem_cache_alloc failed\n"); + return; + } + + /* Copy object */ + so_new->id = so_old->id; + + /* Update references to old object */ + list_del(&so_old->list); + list_add_to_objects(so_new); + + kmem_cache_free(cachep, so_old); + } +} + +static int smo_enable_cache_mobility(int _unused, int __unused) +{ + /* Enable movable objects: BOOM! */ + kmem_cache_setup_mobility(cachep, NULL, smo_cache_migrate); + pr_info("smo: kmem_cache %s defrag enabled\n", SMO_CACHE_NAME); + return 0; +} + +/* + * smo_alloc_objects() - Allocate objects and store reference. + * @nr_objs: Number of objects to allocate. + * @node: NUMA node to allocate objects on. + * + * Allocates @n smo_slub_objects. Stores a reference to them in + * the global list of objects (at the tail of the list). + * + * Return: The number of objects allocated. + */ +static int smo_alloc_objects(int nr_objs, int node) +{ + struct smo_slub_object *so; + int i; + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1; + if (node == SMO_CMD_DEFAUT_ARG) + node = NUMA_NO_NODE; + + for (i = 0; i < nr_objs; i++) { + if (node == NUMA_NO_NODE) + so = kmem_cache_alloc(cachep, GFP_KERNEL); + else + so = kmem_cache_alloc_node(cachep, GFP_KERNEL, node); + if (!so) { + pr_err("smo: Failed to alloc object %d of %d\n", i, nr_objs); + return i; + } + list_add_to_objects(so); + } + return nr_objs; +} + +/* + * smo_free_object() - Frees n objects from position. + * @nr_objs: Number of objects to free. + * @pos: Position in global list to start freeing. + * + * Iterates over the global list of objects to position @pos then frees @n + * objects from there (or to end of list). Does nothing if @n > list length. + * + * Calling with @n==0 frees all objects starting at @pos. + * + * Return: Number of objects freed. + */ +static int smo_free_object(int nr_objs, int pos) +{ + struct smo_slub_object *cur, *tmp; + int deleted = 0; + int i = 0; + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1; + if (pos == SMO_CMD_DEFAUT_ARG) + pos = 0; + + list_for_each_entry_safe(cur, tmp, &objects, list) { + if (i < pos) { + i++; + continue; + } + + list_del(&cur->list); + kmem_cache_free(cachep, cur); + deleted++; + if (deleted == nr_objs) + break; + } + return deleted; +} + +static int index_for_expected_id(long *expected, int size, long id) +{ + int i; + + /* Array is unsorted, just iterate the whole thing */ + for (i = 0; i < size; i++) { + if (expected[i] == id) + return i; + } + return -1; /* Not found */ +} + +static int assert_have_objects(int nr_objs, int keep) +{ + struct smo_slub_object *cur; + long *expected; /* Array of expected IDs */ + int nr_ids; /* Length of array */ + long id; + int index, i; + + nr_ids = nr_objs / keep + 1; + + expected = kmalloc_array(nr_ids, sizeof(long), GFP_KERNEL); + if (!expected) + return -ENOMEM; + + id = 0; + for (i = 0; i < nr_ids; i++) { + expected[i] = id; + id += keep; + } + + list_for_each_entry(cur, &objects, list) { + index = index_for_expected_id(expected, nr_ids, cur->id); + if (index < 0) { + pr_err("smo: ID not found: %ld\n", cur->id); + return -1; + } + + if (expected[index] == -1) { + pr_err("smo: ID already encountered: %ld\n", cur->id); + return -1; + } + expected[index] = -1; + } + return 0; +} + +/* + * smo_run_module_tests() - Runs unit tests from within the module + * @nr_objs: Number of objects to allocate. + * @keep: Free all but 1 in @keep objects. + * + * Allocates @nr_objects then iterates over the allocated objects + * freeing all but 1 out of every @keep objects i.e. for @keep==10 + * keeps the first object then frees the next 9. + * + * Caller is responsible for ensuring that the cache has at most a + * single slab on the partial list without any objects in it. This is + * easy enough to ensure, just call this when the module is freshly + * loaded. + */ +static int smo_run_module_tests(int nr_objs, int keep) +{ + struct smo_slub_object *so; + struct smo_slub_object *cur, *tmp; + long i; + + if (!list_empty(&objects)) { + pr_err("smo: test requires clean module state\n"); + return -1; + } + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1000; + if (keep == SMO_CMD_DEFAUT_ARG) + keep = 10; + + pr_info("smo: test using nr_objs: %d keep: %d\n", nr_objs, keep); + + /* Perhaps we got called like this 'test 1000' */ + if (keep == 0) { + pr_err("Usage: test \n"); + return -1; + } + + /* Test constructor */ + so = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!so) { + pr_err("smo: Failed to alloc object\n"); + return -1; + } + if (so->id != -1) { + pr_err("smo: Initial state incorrect"); + return -1; + } + kmem_cache_free(cachep, so); + + /* + * Test that object migration is correctly implemented by module + * + * This gives us confidence that if new code correctly enables + * object migration (via correct implementation of migrate and + * isolate functions) then the slub allocator code that does + * object migration is correct. + */ + + for (i = 0; i < nr_objs; i++) { + so = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!so) { + pr_err("smo: Failed to alloc object %ld of %d\n", + i, nr_objs); + return -1; + } + so->id = (long)i; + list_add_to_objects(so); + } + + assert_have_objects(nr_objs, 1); + + i = 0; + list_for_each_entry_safe(cur, tmp, &objects, list) { + if (i++ % keep == 0) + continue; + + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + + /* Verify shrink does nothing when migration is not enabled */ + kmem_cache_shrink(cachep); + assert_have_objects(nr_objs, 1); + + /* Now test shrink */ + kmem_cache_setup_mobility(cachep, NULL, smo_cache_migrate); + kmem_cache_shrink(cachep); + /* + * Because of how migrate function deletes and adds objects to + * the objects list we have no way of knowing the order. We + * want to confirm that we have all the objects after shrink + * that we had before we did the shrink. + */ + assert_have_objects(nr_objs, keep); + + /* cleanup */ + list_for_each_entry_safe(cur, tmp, &objects, list) { + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + kmem_cache_shrink(cachep); /* Remove empty slabs from partial list */ + + pr_info("smo: Module tests completed successfully\n"); + return 0; +} + +/* + * struct functions() - Map command to a function pointer. + */ +struct functions { + char *fn_name; + int (*fn_ptr)(int arg0, int arg1); +} fn_tab[] = { + /* + * Because of the way we parse the function table no command + * may have another command as its prefix. + * i.e. this will break: 'foo' and 'foobar' + */ + {"enable", smo_enable_cache_mobility}, + {"alloc", smo_alloc_objects}, + {"free", smo_free_object}, + {"test", smo_run_module_tests}, +}; + +#define FN_TAB_SIZE (sizeof(fn_tab) / sizeof(struct functions)) + +/* + * parse_cmd_buf() - Gets command and arguments command string. + * @buf: Buffer containing the command string. + * @cmd: Out parameter, pointer to the command. + * @arg1: Out parameter, stores the first argument. + * @arg2: Out parameter, stores the second argument. + * + * Parses and tokenizes the input command buffer. Stores a pointer to the + * command (start of @buf) in @cmd. Stores the converted long values for + * argument 1 and 2 in the respective out parameters @arg1 and @arg2. + * + * Since arguments are optional, if they are not found the default values are + * returned. In order for the caller to differentiate defaults from arguments + * of the same value the number of arguments parsed is returned. + * + * Return: Number of arguments found. + */ +static int parse_cmd_buf(char *buf, char **cmd, long *arg1, long *arg2) +{ + int found; + char *ptr; + int ret; + + *arg1 = SMO_CMD_DEFAUT_ARG; + *arg2 = SMO_CMD_DEFAUT_ARG; + found = 0; + + /* Jump over the command, check if there are any args */ + ptr = strsep(&buf, " "); + if (!ptr || !buf) + return found; + + ptr = strsep(&buf, " "); + ret = kstrtol(ptr, 10, arg1); + if (ret < 0) { + pr_err("failed to convert arg, defaulting to %d. (%s)\n", + SMO_CMD_DEFAUT_ARG, ptr); + return found; + } + found++; + if (!buf) /* No second arg */ + return found; + + ptr = strsep(&buf, " "); + ret = kstrtol(ptr, 10, arg2); + if (ret < 0) { + pr_err("failed to convert arg, defaulting to %d. (%s)\n", + SMO_CMD_DEFAUT_ARG, ptr); + return found; + } + found++; + + return found; +} + +/* + * call_function() - Calls the function described by str. + * @str: ' []' + * + * Does table lookup on , calls appropriate function passing + * as a the argument. Optional arg defaults to 1. + */ +static void call_function(char *str) +{ + char *cmd; + long arg1 = 0; + long arg2 = 0; + int i; + + if (!str) + return; + + (void)parse_cmd_buf(str, &cmd, &arg1, &arg2); + + for (i = 0; i < FN_TAB_SIZE; i++) { + char *fn_name = fn_tab[i].fn_name; + + if (strcmp(fn_name, str) == 0) { + fn_tab[i].fn_ptr(arg1, arg2); + return; /* All done */ + } + } + + pr_err("failed to call function for cmd: %s\n", str); +} + +/* + * smo_callfn_debugfs_write() - debugfs write function. + * @file: User file + * @user_buf: Userspace buffer + * @len: Length of the user space buffer + * @off: Offset within the file + * + * Used for triggering functions by writing command to debugfs file. + * + * echo ' ' > /sys/kernel/debug/smo/callfn + * + * Return: Number of bytes copied if request succeeds, + * the corresponding error code otherwise. + */ +static ssize_t smo_callfn_debugfs_write(struct file *file, + const char __user *ubuf, + size_t len, + loff_t *off) +{ + char *kbuf; + int nbytes = 0; + + if (*off != 0 || len == 0) + return -EINVAL; + + kbuf = kzalloc(len, GFP_KERNEL); + if (!kbuf) + return -ENOMEM; + + nbytes = strncpy_from_user(kbuf, ubuf, len); + if (nbytes < 0) + goto out; + + if (kbuf[nbytes - 1] == '\n') + kbuf[nbytes - 1] = '\0'; + + call_function(kbuf); /* Tokenizes kbuf */ +out: + kfree(kbuf); + return nbytes; +} + +const struct file_operations fops_callfn_debugfs = { + .owner = THIS_MODULE, + .write = smo_callfn_debugfs_write, +}; + +static int __init smo_debugfs_init(void) +{ + struct dentry *d; + + smo_debugfs_root = debugfs_create_dir(SMO_DEBUGFS_DIR, NULL); + d = debugfs_create_file("callfn", 0200, smo_debugfs_root, NULL, + &fops_callfn_debugfs); + if (IS_ERR(d)) + return PTR_ERR(d); + + return 0; +} + +static void __exit smo_debugfs_cleanup(void) +{ + debugfs_remove_recursive(smo_debugfs_root); +} + +static int __init smo_cache_init(void) +{ + cachep = kmem_cache_create(SMO_CACHE_NAME, + sizeof(struct smo_slub_object), + 0, 0, smo_object_ctor); + if (!cachep) + return -1; + + return 0; +} + +static void __exit smo_cache_cleanup(void) +{ + struct smo_slub_object *cur, *tmp; + + list_for_each_entry_safe(cur, tmp, &objects, list) { + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + kmem_cache_destroy(cachep); +} + +static int __init smo_init(void) +{ + int ret; + + ret = smo_cache_init(); + if (ret) { + pr_err("smo: Failed to create cache\n"); + return ret; + } + pr_info("smo: Created kmem_cache: %s\n", SMO_CACHE_NAME); + + ret = smo_debugfs_init(); + if (ret) { + pr_err("smo: Failed to init debugfs\n"); + return ret; + } + pr_info("smo: Created debugfs directory: /sys/kernel/debugfs/%s\n", + SMO_DEBUGFS_DIR); + + pr_info("smo: Test module loaded\n"); + return 0; +} +module_init(smo_init); + +static void __exit smo_exit(void) +{ + smo_debugfs_cleanup(); + smo_cache_cleanup(); + + pr_info("smo: Test module removed\n"); +} +module_exit(smo_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Tobin C. Harding"); +MODULE_DESCRIPTION("SLUB Movable Objects test module."); From patchwork Mon Jun 3 04:26:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972221 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF09C1515 for ; Mon, 3 Jun 2019 04:28:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9DBEB28174 for ; Mon, 3 Jun 2019 04:28:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 914D028387; Mon, 3 Jun 2019 04:28:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 67D3E28174 for ; Mon, 3 Jun 2019 04:28:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F89B6B0272; Mon, 3 Jun 2019 00:28:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2A9CA6B0273; Mon, 3 Jun 2019 00:28:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 122216B0274; Mon, 3 Jun 2019 00:28:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id E606D6B0272 for ; Mon, 3 Jun 2019 00:28:34 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id q17so13669883qkc.23 for ; Sun, 02 Jun 2019 21:28:34 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=Phk2yfRAblYf4yx9pTBRgCIG7vqNLM3J9Y9qH9+MD8LAP6QvA7mmRlwo0HQBRx+l9E AUux0+Dm7RYtN3nddv3t7Q1XaEvpJ8sk5Y6cQCeliPfldZS6nvAE7uki9yW48In06d+g vbSqdYmsXEvE4GeuiD39XfmK+xxTr4Ako/686QRqSxLTPKHrdXbM16Fs3wfJ/iQYXGZX PqhJgZGYMxX/eWNYablhkpSbRA5nTlbNmy93rNsYs8LdC9lM9qbRilZHHHFmLHDRlfsk xHhWCz9UAxg3f4MOJzMSMq02+Rw3As357WyzWDDaSSkyJ6nKgj5i1ULwhFA0gdYIkG6M 5FwA== X-Gm-Message-State: APjAAAW69CiRAypm1bnjblY0AfksOJINAzKpxr2iXvr7qd+eOKSctTtN hUz+Cp7WSUiuCWf44LxfQQnig/8V0E2X7Xpp1cn6wPN2YHr5WO4SCHkWNA3I3mNTrzcmKtUcx/z xEsI3QkKfV5n27nCwfmiFwjZqvCwbBp5BVIiEljCobany77GPhuLLTFxPWFFUEeU= X-Received: by 2002:a37:654d:: with SMTP id z74mr1034204qkb.10.1559536114644; Sun, 02 Jun 2019 21:28:34 -0700 (PDT) X-Google-Smtp-Source: APXvYqwNSiJXzLeSnvzGPfB4AylU5SSnaKNk+zHBDZ1YqH4eko+h8SEN2q8eYf+3Gl4fZ7teuWIj X-Received: by 2002:a37:654d:: with SMTP id z74mr1034143qkb.10.1559536112987; Sun, 02 Jun 2019 21:28:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536112; cv=none; d=google.com; s=arc-20160816; b=sJgbWK0pCzbRmJVMYDwAdu0zY9BVV8O+4frm3scrjoKVkWERkM7blOFDcEX1ozPuLR 4t2pxOzmR0fDAKVDBiXeH5sQ6YSdhdTGoG+Q4bHQxSoy6hg5FJd8OJzeYMK5Op8MIB9m dDZoihfM0sNgLSQki450faZI2oHOL0i1g1sU8zyHoHywZHe6ZgVcts42BVn9Vva5deiM csoAs3iaU49lfKfwklEyUIhaLOmld/aby7xdtRZOiIyBQfbBt3Yp0OEFGxTTpEZoS+Ql Y9UKZH/tHr98MYityXqEFEnz7dulxk/DX8nLO0CWuh7SGCiJbH1CCWA7tPT1h/eZTheg wV2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=aXECl08ST0tDcjrb1uc12FIRkCaef9a1hAhCkuWivnSvAQl3TF99Ccaqm5YAMp9fJm pSDLpiNlnH/kfy8n2KsYf5GZSsEI8l7WCzJ16ui1g08oM6r1MddQh7YsECLWD1Rx+C4v i2WU6zc78BQsrxbB8zoH7xD0HBJGtfK3rdSdpOhizh6zg9x1cN/zyo5zszgwqOWQsFK8 REje1foBe7OfaOldnJGkOXXo8BGcHt0b9ofWwfUw5frcl+xKn8FNRcqfhC0z9xsXw3HL QBfwdA3fBgba0gOOZEeB8amcTaeh+B93Fej9GXpstXsWZ895zLS7KnxLSniNW9gU5cpB mmzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=EGrFHcq9; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id 28si2767809qtx.181.2019.06.02.21.28.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:28:32 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=EGrFHcq9; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id A6F5621A9; Mon, 3 Jun 2019 00:28:32 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:28:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=EGrFHcq9 ZnQxczfhgL0BzKQqK3wM5qqM5G52mCbrNqNLAjJp5Pk6Lpyy/6pmCA8qC5BjL40r qv/uC0jH5H1GUtKxWlyw/ZRdf9/8+VjhFr/sdkup9wKcdrbLPA25xV8ClE1XGcat XSgNB3XaySPbeIamorNdHlv2ZQNcukuZHCJOMciSEQB1rFqkFtGE/nB7d6YDpSyo 7NAWC11+Bd0vGrfvQeRMpci9756GFkP0N6Z4wpwXxVIh5Gmr8mUEvBh9IN8mKBKl hD2T1Q0w9GeBlnRPg4T5PbBRPnReeYIT0MqJY4U6W0BEpxrBLZsDaZedOVsHZzNZ nsUgjRStYXv9YQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id 7EC3180063; Mon, 3 Jun 2019 00:28:25 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 08/15] tools/testing/slab: Add object migration test suite Date: Mon, 3 Jun 2019 14:26:30 +1000 Message-Id: <20190603042637.2018-9-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just added a module that enables testing the SLUB allocators ability to defrag/shrink caches via movable objects. Tests are better when they are automated. Add automated testing via a python script for SLUB movable objects. Example output: $ cd path/to/linux/tools/testing/slab $ /slub_defrag.py Please run script as root $ sudo ./slub_defrag.py $ sudo ./slub_defrag.py --debug Loading module ... Slab cache smo_test created Objects per slab: 20 Running sanity checks ... Running module stress test (see dmesg for additional test output) ... Removing module slub_defrag ... Loading module ... Slab cache smo_test created Running test non-movable ... testing slab 'smo_test' prior to enabling movable objects ... verified non-movable slabs are NOT shrinkable Running test movable ... testing slab 'smo_test' after enabling movable objects ... verified movable slabs are shrinkable Removing module slub_defrag ... Signed-off-by: Tobin C. Harding --- tools/testing/slab/slub_defrag.c | 1 + tools/testing/slab/slub_defrag.py | 451 ++++++++++++++++++++++++++++++ 2 files changed, 452 insertions(+) create mode 100755 tools/testing/slab/slub_defrag.py diff --git a/tools/testing/slab/slub_defrag.c b/tools/testing/slab/slub_defrag.c index 4a5c24394b96..8332e69ee868 100644 --- a/tools/testing/slab/slub_defrag.c +++ b/tools/testing/slab/slub_defrag.c @@ -337,6 +337,7 @@ static int smo_run_module_tests(int nr_objs, int keep) /* * struct functions() - Map command to a function pointer. + * If you update this please update the documentation in slub_defrag.py */ struct functions { char *fn_name; diff --git a/tools/testing/slab/slub_defrag.py b/tools/testing/slab/slub_defrag.py new file mode 100755 index 000000000000..41747c0db39b --- /dev/null +++ b/tools/testing/slab/slub_defrag.py @@ -0,0 +1,451 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import subprocess +import sys +from os import path + +# SLUB Movable Objects test suite. +# +# Requirements: +# - CONFIG_SLUB=y +# - CONFIG_SLUB_DEBUG=y +# - The slub_defrag module in this directory. + +# Test SMO using a kernel module that enables triggering arbitrary +# kernel code from userspace via a debugfs file. +# +# Module code is in ./slub_defrag.c, basically the functionality is as +# follows: +# +# - Creates debugfs file /sys/kernel/debugfs/smo/callfn +# - Writes to 'callfn' are parsed as a command string and the function +# associated with command is called. +# - Defines 4 commands (all commands operate on smo_test cache): +# - 'test': Runs module stress tests. +# - 'alloc N': Allocates N slub objects +# - 'free N POS': Frees N objects starting at POS (see below) +# - 'enable': Enables SLUB Movable Objects +# +# The module maintains a list of allocated objects. Allocation adds +# objects to the tail of the list. Free'ing frees from the head of the +# list. This has the effect of creating free slots in the slab. For +# finer grained control over where in the cache slots are free'd POS +# (position) argument may be used. + +# The main() function is reasonably readable; the test suite does the +# following: +# +# 1. Runs the module stress tests. +# 2. Tests the cache without movable objects enabled. +# - Creates multiple partial slabs as explained above. +# - Verifies that partial slabs are _not_ removed by shrink (see below). +# 3. Tests the cache with movable objects enabled. +# - Creates multiple partial slabs as explained above. +# - Verifies that partial slabs _are_ removed by shrink (see below). + +# The sysfs file /sys/kernel/slab//shrink enables calling the +# function kmem_cache_shrink() (see mm/slab_common.c and mm/slub.cc). +# Shrinking a cache attempts to consolidate all partial slabs by moving +# objects if object migration is enable for the cache, otherwise +# shrinking a cache simply re-orders the partial list so as most densely +# populated slab are at the head of the list. + +# Enable/disable debugging output (also enabled via -d | --debug). +debug = False + +# Used in debug messages and when running `insmod`. +MODULE_NAME = "slub_defrag" + +# Slab cache created by the test module. +CACHE_NAME = "smo_test" + +# Set by get_slab_config() +objects_per_slab = 0 +pages_per_slab = 0 +debugfs_mounted = False # Set to true if we mount debugfs. + + +def eprint(*args, **kwargs): + print(*args, file=sys.stderr, **kwargs) + + +def dprint(*args, **kwargs): + if debug: + print(*args, file=sys.stderr, **kwargs) + + +def run_shell(cmd): + return subprocess.call([cmd], shell=True) + + +def run_shell_get_stdout(cmd): + return subprocess.check_output([cmd], shell=True) + + +def assert_root(): + user = run_shell_get_stdout('whoami') + if user != b'root\n': + eprint("Please run script as root") + sys.exit(1) + + +def mount_debugfs(): + mounted = False + + # Check if debugfs is mounted at a known mount point. + ret = run_shell('mount -l | grep /sys/kernel/debug > /dev/null 2>&1') + if ret != 0: + run_shell('mount -t debugfs none /sys/kernel/debug/') + mounted = True + dprint("Mounted debugfs on /sys/kernel/debug") + + return mounted + + +def umount_debugfs(): + dprint("Un-mounting debugfs") + run_shell('umount /sys/kernel/debug') + + +def load_module(): + """Loads the test module. + + We need a clean slab state to start with so module must + be loaded by the test suite. + """ + ret = run_shell('lsmod | grep %s > /dev/null' % MODULE_NAME) + if ret == 0: + eprint("Please unload slub_defrag module before running test suite") + return -1 + + dprint('Loading module ...') + ret = run_shell('insmod %s.ko' % MODULE_NAME) + if ret != 0: # ret==1 on error + return -1 + + dprint("Slab cache %s created" % CACHE_NAME) + return 0 + + +def unload_module(): + ret = run_shell('lsmod | grep %s > /dev/null' % MODULE_NAME) + if ret == 0: + dprint('Removing module %s ...' % MODULE_NAME) + run_shell('rmmod %s > /dev/null 2>&1' % MODULE_NAME) + + +def get_sysfs_value(filename): + """ + Parse slab sysfs files (single line: '20 N0=20') + """ + path = '/sys/kernel/slab/smo_test/%s' % filename + f = open(path, "r") + s = f.readline() + tokens = s.split(" ") + + return int(tokens[0]) + + +def get_nr_objects_active(): + return get_sysfs_value('objects') + + +def get_nr_objects_total(): + return get_sysfs_value('total_objects') + + +def get_nr_slabs_total(): + return get_sysfs_value('slabs') + + +def get_nr_slabs_partial(): + return get_sysfs_value('partial') + + +def get_nr_slabs_full(): + return get_nr_slabs_total() - get_nr_slabs_partial() + + +def get_slab_config(): + """Get relevant information from sysfs.""" + global objects_per_slab + + objects_per_slab = get_sysfs_value('objs_per_slab') + if objects_per_slab < 0: + return -1 + + dprint("Objects per slab: %d" % objects_per_slab) + return 0 + + +def verify_state(nr_objects_active, nr_objects_total, + nr_slabs_partial, nr_slabs_full, nr_slabs_total, msg=''): + err = 0 + got_nr_objects_active = get_nr_objects_active() + got_nr_objects_total = get_nr_objects_total() + got_nr_slabs_partial = get_nr_slabs_partial() + got_nr_slabs_full = get_nr_slabs_full() + got_nr_slabs_total = get_nr_slabs_total() + + if got_nr_objects_active != nr_objects_active: + err = -1 + + if got_nr_objects_total != nr_objects_total: + err = -2 + + if got_nr_slabs_partial != nr_slabs_partial: + err = -3 + + if got_nr_slabs_full != nr_slabs_full: + err = -4 + + if got_nr_slabs_total != nr_slabs_total: + err = -5 + + if err != 0: + dprint("Verify state: %s" % msg) + dprint(" what\t\t\twant\tgot") + dprint("-----------------------------------------") + dprint(" %s\t%d\t%d" % ('nr_objects_active', nr_objects_active, got_nr_objects_active)) + dprint(" %s\t%d\t%d" % ('nr_objects_total', nr_objects_total, got_nr_objects_total)) + dprint(" %s\t%d\t%d" % ('nr_slabs_partial', nr_slabs_partial, got_nr_slabs_partial)) + dprint(" %s\t\t%d\t%d" % ('nr_slabs_full', nr_slabs_full, got_nr_slabs_full)) + dprint(" %s\t%d\t%d\n" % ('nr_slabs_total', nr_slabs_total, got_nr_slabs_total)) + + return err + + +def exec_via_sysfs(command): + ret = run_shell('echo %s > /sys/kernel/debug/smo/callfn' % command) + if ret != 0: + eprint("Failed to echo command to sysfs: %s" % command) + + return ret + + +def enable_movable_objects(): + return exec_via_sysfs('enable') + + +def alloc(n): + exec_via_sysfs("alloc %d" % n) + + +def free(n, pos = 0): + exec_via_sysfs('free %d %d' % (n, pos)) + + +def shrink(): + ret = run_shell('slabinfo smo_test -s') + if ret != 0: + eprint("Failed to execute slabinfo -s") + + +def sanity_checks(): + # Verify everything is 0 to start with. + return verify_state(0, 0, 0, 0, 0, "sanity check") + + +def test_non_movable(): + one_over = objects_per_slab + 1 + + dprint("testing slab 'smo_test' prior to enabling movable objects ...") + + alloc(one_over) + + objects_active = one_over + objects_total = objects_per_slab * 2 + slabs_partial = 1 + slabs_full = 1 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: initial allocation") + if ret != 0: + eprint("test_non_movable: failed to verify initial state") + return -1 + + # Free object from first slot of first slab. + free(1) + objects_active = one_over - 1 + objects_total = objects_per_slab * 2 + slabs_partial = 2 + slabs_full = 0 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: after free") + if ret != 0: + eprint("test_non_movable: failed to verify after free") + return -1 + + # Non-movable cache, shrink should have no effect. + shrink() + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: after shrink") + if ret != 0: + eprint("test_non_movable: failed to verify after shrink") + return -1 + + # Cleanup + free(objects_per_slab) + shrink() + + dprint("verified non-movable slabs are NOT shrinkable") + return 0 + + +def test_movable(): + one_over = objects_per_slab + 1 + + dprint("testing slab 'smo_test' after enabling movable objects ...") + + alloc(one_over) + + objects_active = one_over + objects_total = objects_per_slab * 2 + slabs_partial = 1 + slabs_full = 1 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: initial allocation") + if ret != 0: + eprint("test_movable: failed to verify initial state") + return -1 + + # Free object from first slot of first slab. + free(1) + objects_active = one_over - 1 + objects_total = objects_per_slab * 2 + slabs_partial = 2 + slabs_full = 0 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: after free") + if ret != 0: + eprint("test_movable: failed to verify after free") + return -1 + + # movable cache, shrink should move objects and free slab. + shrink() + objects_active = one_over - 1 + objects_total = objects_per_slab * 1 + slabs_partial = 0 + slabs_full = 1 + slabs_total = 1 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: after shrink") + if ret != 0: + eprint("test_movable: failed to verify after shrink") + return -1 + + # Cleanup + free(objects_per_slab) + shrink() + + dprint("verified movable slabs are shrinkable") + return 0 + + +def dprint_start_test(test): + dprint("Running %s ..." % test) + + +def dprint_done(): + dprint("") + + +def run_test(fn, desc): + dprint_start_test(desc) + ret = fn() + if ret < 0: + fail_test(desc) + dprint_done() + + +# Load and unload the module for this test to ensure clean state. +def run_module_stress_test(): + dprint("Running module stress test (see dmesg for additional test output) ...") + + unload_module() + ret = load_module() + if ret < 0: + cleanup_and_exit(ret) + + exec_via_sysfs("test"); + + unload_module() + + dprint() + + +def fail_test(msg): + eprint("\nFAIL: test failed: '%s' ... aborting\n" % msg) + cleanup_and_exit(1) + + +def display_help(): + print("Usage: %s [OPTIONS]\n" % path.basename(sys.argv[0])) + print("\tRuns defrag test suite (a.k.a. SLUB Movable Objects)\n") + print("OPTIONS:") + print("\t-d | --debug Enable verbose debug output") + print("\t-h | --help Print this help and exit") + + +def cleanup_and_exit(return_code): + global debugfs_mounted + + if debugfs_mounted == True: + umount_debugfs() + + unload_module() + + sys.exit(return_code) + + +def main(): + global debug + + if len(sys.argv) > 1: + if sys.argv[1] == '-h' or sys.argv[1] == '--help': + display_help() + sys.exit(0) + + if sys.argv[1] == '-d' or sys.argv[1] == '--debug': + debug = True + + assert_root() + + # Use cleanup_and_exit() instead of sys.exit() after mounting debugfs. + debugfs_mounted = mount_debugfs() + + # Loads and unloads the module. + run_module_stress_test() + + ret = load_module() + if (ret < 0): + cleanup_and_exit(ret) + + ret = get_slab_config() + if (ret != 0): + fail_test("get slab config details") + + run_test(sanity_checks, "sanity checks") + + run_test(test_non_movable, "test non-movable") + + ret = enable_movable_objects() + if (ret != 0): + fail_test("enable movable objects") + + run_test(test_movable, "test movable") + + cleanup_and_exit(0) + +if __name__== "__main__": + main() From patchwork Mon Jun 3 04:26:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972225 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 14DF21515 for ; Mon, 3 Jun 2019 04:28:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0431828174 for ; Mon, 3 Jun 2019 04:28:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ECA3F28387; Mon, 3 Jun 2019 04:28:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 360C628385 for ; Mon, 3 Jun 2019 04:28:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E43986B026B; Mon, 3 Jun 2019 00:28:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E197C6B0273; Mon, 3 Jun 2019 00:28:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D0B716B0274; Mon, 3 Jun 2019 00:28:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id AF1E26B026B for ; Mon, 3 Jun 2019 00:28:41 -0400 (EDT) Received: by mail-qt1-f199.google.com with SMTP id 97so6490247qtb.16 for ; Sun, 02 Jun 2019 21:28:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=nu/o7ND31/Jz1+fQrVDSXNqvU8Wx++J3ZYdTw/+7V9o=; b=GXV1RuRkGo1VXKVmU4K2OStf3A8+Tm+AyOLdbh3E6rd2lkmaA5R0DjfLpklrhGACKN ZuxoGfaBgQTRpixHy9GbbP6zrxltTmPBitNTsUNW5+fIxix3ylMrylKmvurviI8obmfF Ot6LoLwATOVKTLRyThvPSCbbqHt85E5PEwDEPqNsoFqh2ZCGRGifH4W5ckkVqDud1FKY 0zUPwybo6YUH8EaJJqj3QHpcke9Pufp/cNSEsRRKv3zXKNJLJ9jQnaYlNVGbmPcz2bDw tJGNxLywfuT4mT3uXpkAGwR+zFUHFSbY63BWjTBQ8/LbbIdZR5lOvPYKh9ab5tESMdJO 4r3g== X-Gm-Message-State: APjAAAXK70SW+AZFeR4fYlOci9YuB35FPEHhVBT7ivwYMbReF8Rkt1ml hQYfQk/SUogaVCLXtZggePYCfY50IElAghBbZ2kbEwd6Bau9J7EsJYRJWaxLdAy8GHfMiuAC7FN SXzc1/KgDt9zJ/TwX2GUtro0m22TF6A5Kaukhc1/U+A3JLpCsyKlquiIAuOCd5pE= X-Received: by 2002:aed:24d9:: with SMTP id u25mr21507343qtc.111.1559536121465; Sun, 02 Jun 2019 21:28:41 -0700 (PDT) X-Google-Smtp-Source: APXvYqy7jWVpzp0tF3a6eS5YNbnJVOMZYfN5mLMQaBjP46ILTMjnu1+DeLCFuZwXM0eNu69U5c93 X-Received: by 2002:aed:24d9:: with SMTP id u25mr21507297qtc.111.1559536120148; Sun, 02 Jun 2019 21:28:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536120; cv=none; d=google.com; s=arc-20160816; b=FYJ2qrkoJ5c8+xQPumwU14816Ur7W3916ZrlRaWGV1gkgvjejvbrMXiAY6I59OhdIp C1wS2Nsw7Rg7S8pWnZ4NTcUd17KofqCyn+0vGF6wnNzgvmXUX7ISpVTWbqu0ZoY+2EHx MPd68iWUpMxcN4ki1cTl6zSYCfVwa8XSNkDMQuVv/JnuIZGI+OLRdg6kU33ssRQdNBGw 8qEYKGC0Qsxh2t2hTRss2nnoNwtTy03EUwvqlYwhxYKYeMz1DWbld7uLKL94hLIFFGna ABFFqvXReSaH8HU1trgaa5Hr31IA5lEHe2IjT7weYHhQc88Kv0AyFYg+99Zv1AJx2kl2 VARA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=nu/o7ND31/Jz1+fQrVDSXNqvU8Wx++J3ZYdTw/+7V9o=; b=MuxfwLjDZtVL8SMwUdOHsCmWnBlFfxhPvq11iG/ONTXojPtufRhYNg5dKj5QFYYOA3 v/0NrkkWaWVbCZrnKlrWraBAPlNabm3zgox9lz2UkXf9TiAJiJEi/yDKqkGosmz1xZmq UxJGVoSTdgRtf1d2OY5NrZSXmULho8wntW3HlZrPt7QW+EoJqpNCLn+srS5bJg4pzOqj cSayJ/txh/RZBL/LIlU1szPdZty5YHdyiJbJwrSqhMhucZc86PFGBsd9wM14s8n2/Jh7 LLiix/wGdfPb93HP1RzYOyg5Q+BY4dStVzu5vCZgDxepZ4V39lpSORRfN3YQjZEu9END N60w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=zBRQziWx; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id p57si5058329qtc.217.2019.06.02.21.28.40 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:28:40 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=zBRQziWx; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id E2EE61283; Mon, 3 Jun 2019 00:28:39 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:28:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=nu/o7ND31/Jz1+fQrVDSXNqvU8Wx++J3ZYdTw/+7V9o=; b=zBRQziWx nrqfLlLsLvXdwhD7MQ4bSso9ajZt8TG1PQfeRTZ8f/5YkjvRkNw1u8SgbCmrP2x1 6W5DFPIFmI1Pu5rPZns4vcBrQgsFzku9TKaSyY6Ov5M5+3n6IRnZCetxzrSMG5z2 EF2RrLCnxsG5WjducoH6igzGN9m4sdSFHNtdlq8TLILts+usAC0RSBpN29uuq2Fq mpG9998/sFqKmTAoqY3U0i0BsKvP6vtHOnO8nqt90CU5l4ENZBIgXPDo2NvgO9eo jey7ALOf/Vx+RnqDl2AbHIML1JpeeLMoKGqXccVoz8JrH+ADYKOvebDoxUrW9UvL xCXOrFixi9lxCQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id B5F6D80062; Mon, 3 Jun 2019 00:28:32 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 09/15] lib: Separate radix_tree_node and xa_node slab cache Date: Mon, 3 Jun 2019 14:26:31 +1000 Message-Id: <20190603042637.2018-10-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Earlier, Slab Movable Objects (SMO) was implemented. The XArray is now able to take advantage of SMO in order to make xarray nodes movable (when using the SLUB allocator). Currently the radix tree uses the same slab cache as the XArray. Only XArray nodes are movable _not_ radix tree nodes. We can give the radix tree its own slab cache to overcome this. In preparation for implementing XArray object migration (xa_node objects) via Slab Movable Objects add a slab cache solely for XArray nodes and make the XArray use this slab cache instead of the radix_tree_node slab cache. Cc: Matthew Wilcox Signed-off-by: Tobin C. Harding --- include/linux/xarray.h | 3 +++ init/main.c | 2 ++ lib/radix-tree.c | 2 +- lib/xarray.c | 48 ++++++++++++++++++++++++++++++++++-------- 4 files changed, 45 insertions(+), 10 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index 0e01e6129145..773f91f8e1db 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -42,6 +42,9 @@ #define BITS_PER_XA_VALUE (BITS_PER_LONG - 1) +/* Called from init/main.c */ +void xarray_slabcache_init(void); + /** * xa_mk_value() - Create an XArray entry from an integer. * @v: Value to store in XArray. diff --git a/init/main.c b/init/main.c index 66a196c5e4c3..8c409a5dc937 100644 --- a/init/main.c +++ b/init/main.c @@ -107,6 +107,7 @@ static int kernel_init(void *); extern void init_IRQ(void); extern void radix_tree_init(void); +extern void xarray_slabcache_init(void); /* * Debug helper: via this flag we know that we are in 'early bootup code' @@ -622,6 +623,7 @@ asmlinkage __visible void __init start_kernel(void) "Interrupts were enabled *very* early, fixing it\n")) local_irq_disable(); radix_tree_init(); + xarray_slabcache_init(); /* * Set up housekeeping before setting up workqueues to allow the unbound diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 18c1dfbb1765..e6127c4c84b5 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -31,7 +31,7 @@ /* * Radix tree node cache. */ -struct kmem_cache *radix_tree_node_cachep; +static struct kmem_cache *radix_tree_node_cachep; /* * The radix tree is variable-height, so an insert operation not only has diff --git a/lib/xarray.c b/lib/xarray.c index 6be3acbb861f..861c042daa1d 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -27,6 +27,8 @@ * @entry refers to something stored in a slot in the xarray */ +static struct kmem_cache *xa_node_cachep; + static inline unsigned int xa_lock_type(const struct xarray *xa) { return (__force unsigned int)xa->xa_flags & 3; @@ -244,9 +246,21 @@ void *xas_load(struct xa_state *xas) } EXPORT_SYMBOL_GPL(xas_load); -/* Move the radix tree node cache here */ -extern struct kmem_cache *radix_tree_node_cachep; -extern void radix_tree_node_rcu_free(struct rcu_head *head); +static void xa_node_rcu_free(struct rcu_head *head) +{ + struct xa_node *node = container_of(head, struct xa_node, rcu_head); + + /* + * Must only free zeroed nodes into the slab. We can be left with + * non-NULL entries by radix_tree_free_nodes, so clear the entries + * and tags here. + */ + memset(node->slots, 0, sizeof(node->slots)); + memset(node->tags, 0, sizeof(node->tags)); + INIT_LIST_HEAD(&node->private_list); + + kmem_cache_free(xa_node_cachep, node); +} #define XA_RCU_FREE ((struct xarray *)1) @@ -254,7 +268,7 @@ static void xa_node_free(struct xa_node *node) { XA_NODE_BUG_ON(node, !list_empty(&node->private_list)); node->array = XA_RCU_FREE; - call_rcu(&node->rcu_head, radix_tree_node_rcu_free); + call_rcu(&node->rcu_head, xa_node_rcu_free); } /* @@ -270,7 +284,7 @@ static void xas_destroy(struct xa_state *xas) if (!node) return; XA_NODE_BUG_ON(node, !list_empty(&node->private_list)); - kmem_cache_free(radix_tree_node_cachep, node); + kmem_cache_free(xa_node_cachep, node); xas->xa_alloc = NULL; } @@ -298,7 +312,7 @@ bool xas_nomem(struct xa_state *xas, gfp_t gfp) xas_destroy(xas); return false; } - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc(xa_node_cachep, gfp); if (!xas->xa_alloc) return false; XA_NODE_BUG_ON(xas->xa_alloc, !list_empty(&xas->xa_alloc->private_list)); @@ -327,10 +341,10 @@ static bool __xas_nomem(struct xa_state *xas, gfp_t gfp) } if (gfpflags_allow_blocking(gfp)) { xas_unlock_type(xas, lock_type); - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc(xa_node_cachep, gfp); xas_lock_type(xas, lock_type); } else { - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc(xa_node_cachep, gfp); } if (!xas->xa_alloc) return false; @@ -358,7 +372,7 @@ static void *xas_alloc(struct xa_state *xas, unsigned int shift) if (node) { xas->xa_alloc = NULL; } else { - node = kmem_cache_alloc(radix_tree_node_cachep, + node = kmem_cache_alloc(xa_node_cachep, GFP_NOWAIT | __GFP_NOWARN); if (!node) { xas_set_err(xas, -ENOMEM); @@ -1971,6 +1985,22 @@ void xa_destroy(struct xarray *xa) } EXPORT_SYMBOL(xa_destroy); +static void xa_node_ctor(void *arg) +{ + struct xa_node *node = arg; + + memset(node, 0, sizeof(*node)); + INIT_LIST_HEAD(&node->private_list); +} + +void __init xarray_slabcache_init(void) +{ + xa_node_cachep = kmem_cache_create("xarray_node", + sizeof(struct xa_node), 0, + SLAB_PANIC | SLAB_RECLAIM_ACCOUNT, + xa_node_ctor); +} + #ifdef XA_DEBUG void xa_dump_node(const struct xa_node *node) { From patchwork Mon Jun 3 04:26:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972229 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4BBF814DB for ; Mon, 3 Jun 2019 04:28:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B5CA28174 for ; Mon, 3 Jun 2019 04:28:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2F1EB28387; Mon, 3 Jun 2019 04:28:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FB1A28174 for ; Mon, 3 Jun 2019 04:28:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8EB526B0274; Mon, 3 Jun 2019 00:28:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8C1AD6B0276; Mon, 3 Jun 2019 00:28:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 78BCA6B0275; Mon, 3 Jun 2019 00:28:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 5A3546B026C for ; Mon, 3 Jun 2019 00:28:49 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id r58so6515748qtb.5 for ; Sun, 02 Jun 2019 21:28:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=rxVuyzm1WS3UxHiswMwmyEkW+N4rdmsFgqzw5exGUx0=; b=HS8qT8xPOLygp5emzcVhDJz0DP9oIiF2vy2NARWDSEtkUogH/DEZpsGtIjHG4QVh8S ioyBUSiXig7XRuNaqK2q8SjzseUweaFzqvObUi1cTJELHnjJuL4lNBOKtTgBXf6HaGyo BZtrScjWlcBw26f7mqCQFB8twuNJ59QPnwvsu3DmMP6V8PTwbdDJqliycb1XWdEfzz2H 9o7W0FJu/1UiSHo6bQNxRfLzE5rcMl6TSpWkoDsjoRz4B2kkjCQGIxayH+NOh2yC2wZ5 Z2UoBDqL70iDigkfIvLAWUzkuZ5kjXJQaGfy+VXgRuuIxTi5a/H8QLCQ+WeQrTcU1XbO 7ouw== X-Gm-Message-State: APjAAAUH5wJDHrzvKRUNKJ8qF/L5EmYNp+cIGHunn3I+UJjtNsimNP9G IFMIvEpTY5eEEtEtDmtJ9GLo0cjdEc2MT5kuqw7V/jiXm1xorel+PHBSVnJSRaWXXFTPrgdtFOT 8dWXgFGS3xGbOftMPuSa8wQJliQXx8WbRl5O1gUxMkJgl04LOfBaLcRcyqRUgpMY= X-Received: by 2002:a05:620a:247:: with SMTP id q7mr17246262qkn.265.1559536129140; Sun, 02 Jun 2019 21:28:49 -0700 (PDT) X-Google-Smtp-Source: APXvYqw0nKM8777HlEjLT3735Zs7ScZrEVv1c9o4tS+z0IdcfCTmG5F67A5lzN+tcLqEI4i+GX+2 X-Received: by 2002:a05:620a:247:: with SMTP id q7mr17246221qkn.265.1559536128130; Sun, 02 Jun 2019 21:28:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536128; cv=none; d=google.com; s=arc-20160816; b=prOU82fCeaeuSycXMcjivCx6+wGWErs0PwfL5XmG7Y+7W3mquW50z+jK+coGm2fwmK ehQsF8JwcoMw0H2lgVjhkoJN5L5bp5x6KgVvVzul82v1YwbxVw8AO4gZ4jufU6G6S8jW YwwtF3elsFtHlN3qxH1WhlfaHXoXLPdx7b/lTFfXp/lkJN7lPWa5t6CDbqy5aWVfVQIP DDx9ZmZ1VzynFRrpQ+dwIaW0wQ2Y+rFP9Lzw9wS/6J43cT9tZLX6GhVTI7FmujZx1ZBA nuBp3MYlm+2Neqji7Lfjm2y/xUmcDPYq1JbVWbKO8F0KrV63sTgAV5Gf2pF1brKJd4HX o69Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=rxVuyzm1WS3UxHiswMwmyEkW+N4rdmsFgqzw5exGUx0=; b=Khhk9IBIAeBD6I00pMy2u/7pKqtFlHZRlVRr9mi0NkH5/JPsTDzDqYGBof6hTUr+s0 corE1pRETqKer2wuDwzdAVNjhE7BjnMUwKJyyoEi+am2ZR5ijy3SSEhkgOglNzGCghh4 84yZ4NcGdWRVwH1k3dHFnw2h74TqL27FxPMo9afoR7T4cjG8Yuvkpp6tHsjOddKPUq3K yaOxcIXFkom3O9GGbGTzREK9h1fUyJBU1VVIjNESvdKsfl/BQJSYK7wqN10X8xKwlQFw Rzic3JDIzVI5XrXN7IkMjjynXfIFiMFnxa2haN0/OBtI48VjVT+SdcpONGBERVH9OtPT whcg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=3d+ZatUl; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id z193si8540868qka.236.2019.06.02.21.28.48 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:28:48 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=3d+ZatUl; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 826751320; Mon, 3 Jun 2019 00:28:47 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:28:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=rxVuyzm1WS3UxHiswMwmyEkW+N4rdmsFgqzw5exGUx0=; b=3d+ZatUl A1wPrysCoCMns7irCJnO7Xzs7tZnZQVOKKKVaWHjLzAFoXe4z13qk76ntzD8JLfU 4HaxCNIuJL/exRpJajmfocUXwv4Ch6gFZ4YYN09t0niw059zAsAjvYo522x8vOov 8L9UEKPmAyncQ0HRHJyMyevUkW+tmjAj1SZXGxDL2hHU/7DRwv7nvFU5FVNe4xNv /uneC2szDjRVXRGohIKjBhyC3shW1yigoMlax+huURJH4w+iuGc2SXRa0N6c6PkO RsV4+o1I9wI13p7aJyXEBTIcn0oPteBIq+ELuOTKKcOimm1Bu1VI1dQi3E9mnaU5 LqbFoNczAeDp/A== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id EC9598005C; Mon, 3 Jun 2019 00:28:39 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 10/15] xarray: Implement migration function for xa_node objects Date: Mon, 3 Jun 2019 14:26:32 +1000 Message-Id: <20190603042637.2018-11-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Recently Slab Movable Objects (SMO) was implemented for the SLUB allocator. The XArray can take advantage of this and make the xa_node slab cache objects movable. Implement functions to migrate objects and activate SMO when we initialise the XArray slab cache. This is based on initial code by Matthew Wilcox and was modified to work with slab object migration. Cc: Matthew Wilcox Signed-off-by: Tobin C. Harding --- lib/xarray.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/lib/xarray.c b/lib/xarray.c index 861c042daa1d..9354e0f01f26 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1993,12 +1993,73 @@ static void xa_node_ctor(void *arg) INIT_LIST_HEAD(&node->private_list); } +static void xa_object_migrate(struct xa_node *node, int numa_node) +{ + struct xarray *xa = READ_ONCE(node->array); + void __rcu **slot; + struct xa_node *new_node; + int i; + + /* Freed or not yet in tree then skip */ + if (!xa || xa == XA_RCU_FREE) + return; + + new_node = kmem_cache_alloc_node(xa_node_cachep, GFP_KERNEL, numa_node); + if (!new_node) { + pr_err("%s: slab cache allocation failed\n", __func__); + return; + } + + xa_lock_irq(xa); + + /* Check again..... */ + if (xa != node->array) { + node = new_node; + goto unlock; + } + + memcpy(new_node, node, sizeof(struct xa_node)); + + if (list_empty(&node->private_list)) + INIT_LIST_HEAD(&new_node->private_list); + else + list_replace(&node->private_list, &new_node->private_list); + + for (i = 0; i < XA_CHUNK_SIZE; i++) { + void *x = xa_entry_locked(xa, new_node, i); + + if (xa_is_node(x)) + rcu_assign_pointer(xa_to_node(x)->parent, new_node); + } + if (!new_node->parent) + slot = &xa->xa_head; + else + slot = &xa_parent_locked(xa, new_node)->slots[new_node->offset]; + rcu_assign_pointer(*slot, xa_mk_node(new_node)); + +unlock: + xa_unlock_irq(xa); + xa_node_free(node); + rcu_barrier(); +} + +static void xa_migrate(struct kmem_cache *s, void **objects, int nr, + int node, void *_unused) +{ + int i; + + for (i = 0; i < nr; i++) + xa_object_migrate(objects[i], node); +} + void __init xarray_slabcache_init(void) { xa_node_cachep = kmem_cache_create("xarray_node", sizeof(struct xa_node), 0, SLAB_PANIC | SLAB_RECLAIM_ACCOUNT, xa_node_ctor); + + kmem_cache_setup_mobility(xa_node_cachep, NULL, xa_migrate); } #ifdef XA_DEBUG From patchwork Mon Jun 3 04:26:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972233 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 490CF1515 for ; Mon, 3 Jun 2019 04:28:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 37A9928174 for ; Mon, 3 Jun 2019 04:28:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2B49D28387; Mon, 3 Jun 2019 04:28:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4897F28174 for ; Mon, 3 Jun 2019 04:28:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 25BD36B026D; Mon, 3 Jun 2019 00:28:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 20C9E6B0275; Mon, 3 Jun 2019 00:28:57 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FA416B0276; Mon, 3 Jun 2019 00:28:57 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id E1EF46B026D for ; Mon, 3 Jun 2019 00:28:56 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id w184so13734147qka.15 for ; Sun, 02 Jun 2019 21:28:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=Sno8mMpsaF29Abi4GiGRhuSU1U1kDlNDoa52iU9Af2iH32wgB14OXVG4Pm1jXiruPQ 6LBOYhNM4J1mvHWfbiraicx2KnLO4FTbLIP1Hq6S2uJFVO/ltXtzD+lobMA2WZ41+FDZ mK9nsU3Rg5SUzuVsFQ6Tf9fSRx7GsXXN5pj0tK6XQbzxQf/xIJJeVQTgeF+juujfiLMY 44PWxi0tB7UFz4LPKqhfgwsh63QzIGcOKUHZaU3dohOHGbGbIFc15Wet5YZ3Sewwlqed T7QDAmcE7jZQeejKfXhAl6D1qwUdBBUjBK44SQ93H5gv95CX3wDE34CVxE8+OdOccUFu 3caQ== X-Gm-Message-State: APjAAAVWeh5I9DatScGjPlYzPxZiWbmBRte/0YhoRqm82OmV/s1joVUS nMJBtlzZ9di9G/ztzR0vFNBiplslSCehtth2v+A71JJ0pmdIzRzTEfFPgbvJNpyE0GfgZCRogPR BVoidRCToDlYWIdSGVftKJ8+kPpIMj8B2kGwIc0fkQrOwGd5JX2K58s28EFaoCIw= X-Received: by 2002:ac8:66d8:: with SMTP id m24mr21051666qtp.355.1559536136657; Sun, 02 Jun 2019 21:28:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqy1LZbY8j8CPtbPfYHhEq4arnCRcfAW4hcF7oyKYe567gNNYURBhW2mngDCqBHVquAnAVAK X-Received: by 2002:ac8:66d8:: with SMTP id m24mr21051607qtp.355.1559536135304; Sun, 02 Jun 2019 21:28:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536135; cv=none; d=google.com; s=arc-20160816; b=VTE3nCWnryJdopAgzu6PiyGfe/rdQ8JDgl5333oh1EprOHkKlyV0XPVwot08clrBC3 r6WzERJTco9FVZt3KtWPd/v2RBiilDA9Eg60LEKUNIwdsKfC7n2rseUa7ja+OogcyLVF xdA49YQCEZaZDQhWic7RT6hjWVWSfp6uWbiRl+RACEC/cKZ4yloLJb1HDesM47u0zISp BV11pWIug+AllSa/V3o5NE4yc1oafMn9FHmLmDgKFfs2ngsfVO/b4QfOcqjKmOTjjp1G cQG9a+KWTwpVRWqovxZ9EmJV/P47q1gOc6KNh/OujI2TMVS1hh6Y6t1WTnkndAC8nlvA LwVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=DniM7aHOz4ieG4HfcBTOwQzot1rPriIotN7gUD8K0cd4354P80Kluv9D9EiM1PzuyI HAyNqRxP/ylF1aLzb2cbAa//9QR0PrY9scB9uRagrqtb7M08k/zBhvR372D308jWezSh 60+RcArbriDUwRNNGc5mAYamU37eldlWS/8XS6CUbcPDFRCTLTOuCZrZJTmY88l8snzA skj1uETYassFK+XdXwnIggvNLWVnAcpg/FckRj7YgRyneKytdAsPToU/7+hVOY+g8D9O YxYpBXRwh75XIhHRJzgYCURaJQ3OfRA6MJMVpTzTOEyr7Sqd78024IG2puQ+7NoZmMdk DOyQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="Vd/2ekDx"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id e2si1545302qvh.40.2019.06.02.21.28.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:28:55 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="Vd/2ekDx"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id CC8E921F6; Mon, 3 Jun 2019 00:28:54 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:28:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=Vd/2ekDx gA5d3InCQyT99wEZUUfJWZne58P6Esdwcw/xM133umlva+wJNXyfPoz2p+X2nSVa Yg1NhtRjslyxJkbQuZnNryJf81LZ0/0IeWTBoTLYw5wFCWJjxCMxn0tSCUxRdAGr LV9ouS97aXG/AD4jJk+jXfnUJEbBuyrDoCTCtbZzniHLECz4yxn0Z3WGdCxF84Qu tvKZvBze+IlaeIxYHaQIAoMQcJryg+uHmPf8in7tAINrOw3wKoAOOR8XIrjq1rgC fSqDli/Srm+vV15B19D/vuzv48DHAMsyBO2dR83DRPr4s11595yYCh4OR7j1spE/ 4dNpA4sThJ5MLA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id A2AC780061; Mon, 3 Jun 2019 00:28:47 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 11/15] tools/testing/slab: Add XArray movable objects tests Date: Mon, 3 Jun 2019 14:26:33 +1000 Message-Id: <20190603042637.2018-12-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just implemented movable objects for the XArray. Let's test it intree. Add test module for the XArray's movable objects implementation. Functionality of the XArray Slab Movable Object implementation can usually be seen by simply by using `slabinfo` on a running machine since the radix tree is typically in use on a running machine and will have partial slabs. For repeated testing we can use the test module to run to simulate a workload on the XArray then use `slabinfo` to test object migration is functioning. If testing on freshly spun up VM (low radix tree workload) it may be necessary to load/unload the module a number of times to create partial slabs. Example test session -------------------- Relevant /proc/slabinfo column headers: name Prior to testing slabinfo report for radix_tree_node: # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8352 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 497 Sanity Checks : On Total: 8142848 SlabObj: 912 Full : 473 Redzoning : On Used : 4810752 SlabSiz: 16384 Partial: 24 Poisoning : On Loss : 3332096 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2806272 Align : 8 Objects: 17 Tracing : Off Lpadd: 437360 Here you can see the kernel was built with Slab Movable Objects enabled for the XArray (XArray uses the radix tree below the surface). After inserting the test module (note we have triggered allocation of a number of radix tree nodes increasing the object count but decreasing the number of partial slabs): # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8442 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 499 Sanity Checks : On Total: 8175616 SlabObj: 912 Full : 484 Redzoning : On Used : 4862592 SlabSiz: 16384 Partial: 15 Poisoning : On Loss : 3313024 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2836512 Align : 8 Objects: 17 Tracing : Off Lpadd: 439120 Now we can shrink the radix_tree_node cache: # slabinfo radix_tree_node --shrink # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8515 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 501 Sanity Checks : On Total: 8208384 SlabObj: 912 Full : 500 Redzoning : On Used : 4904640 SlabSiz: 16384 Partial: 1 Poisoning : On Loss : 3303744 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2861040 Align : 8 Objects: 17 Tracing : Off Lpadd: 440880 Note the single remaining partial slab. Signed-off-by: Tobin C. Harding --- tools/testing/slab/Makefile | 2 +- tools/testing/slab/slub_defrag_xarray.c | 211 ++++++++++++++++++++++++ 2 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 tools/testing/slab/slub_defrag_xarray.c diff --git a/tools/testing/slab/Makefile b/tools/testing/slab/Makefile index 440c2e3e356f..44c18d9a4d52 100644 --- a/tools/testing/slab/Makefile +++ b/tools/testing/slab/Makefile @@ -1,4 +1,4 @@ -obj-m += slub_defrag.o +obj-m += slub_defrag.o slub_defrag_xarray.o KTREE=../../.. diff --git a/tools/testing/slab/slub_defrag_xarray.c b/tools/testing/slab/slub_defrag_xarray.c new file mode 100644 index 000000000000..41143f73256c --- /dev/null +++ b/tools/testing/slab/slub_defrag_xarray.c @@ -0,0 +1,211 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SMOX_CACHE_NAME "smox_test" +static struct kmem_cache *cachep; + +/* + * Declare XArrays globally so we can clean them up on module unload. + */ + +/* Used by test_smo_xarray()*/ +DEFINE_XARRAY(things); + +/* Thing to store pointers to in the XArray */ +struct smox_thing { + long id; +}; + +/* It's up to the caller to ensure id is unique */ +static struct smox_thing *alloc_thing(int id) +{ + struct smox_thing *thing; + + thing = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!thing) + return ERR_PTR(-ENOMEM); + + thing->id = id; + return thing; +} + +/** + * smox_object_ctor() - SMO object constructor function. + * @ptr: Pointer to memory where the object should be constructed. + */ +void smox_object_ctor(void *ptr) +{ + struct smox_thing *thing = ptr; + + thing->id = -1; +} + +/** + * smox_cache_migrate() - kmem_cache migrate function. + * @cp: kmem_cache pointer. + * @objs: Array of pointers to objects to migrate. + * @size: Number of objects in @objs. + * @node: NUMA node where the object should be allocated. + * @private: Pointer returned by kmem_cache_isolate_func(). + */ +void smox_cache_migrate(struct kmem_cache *cp, void **objs, int size, + int node, void *private) +{ + struct smox_thing **ptrs = (struct smox_thing **)objs; + struct smox_thing *old, *new; + struct smox_thing *thing; + unsigned long index; + void *entry; + int i; + + for (i = 0; i < size; i++) { + old = ptrs[i]; + + new = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!new) { + pr_debug("kmem_cache_alloc failed\n"); + return; + } + + new->id = old->id; + + /* Update reference the brain dead way */ + xa_for_each(&things, index, thing) { + if (thing == old) { + entry = xa_store(&things, index, new, GFP_KERNEL); + if (entry != old) { + pr_err("failed to exchange new/old\n"); + return; + } + } + } + kmem_cache_free(cachep, old); + } +} + +/* + * test_smo_xarray() - Run some tests using an XArray. + */ +static int test_smo_xarray(void) +{ + const int keep = 6; /* Free 5 out of 6 items */ + const int nr_items = 10000; + struct smox_thing *thing; + unsigned long index; + void *entry; + int expected; + int i; + + /* + * Populate XArray, this adds to the radix_tree_node cache as + * well as the smox_test cache. + */ + for (i = 0; i < nr_items; i++) { + thing = alloc_thing(i); + entry = xa_store(&things, i, thing, GFP_KERNEL); + if (xa_is_err(entry)) { + pr_err("smox: failed to allocate entry: %d\n", i); + return -ENOMEM; + } + } + + /* Now free items, putting holes in both caches. */ + for (i = 0; i < nr_items; i++) { + if (i % keep == 0) + continue; + + thing = xa_erase(&things, i); + if (xa_is_err(thing)) + pr_err("smox: error erasing entry: %d\n", i); + kmem_cache_free(cachep, thing); + } + + expected = 0; + xa_for_each(&things, index, thing) { + if (thing->id != expected || index != expected) { + pr_err("smox: error; got %ld want %d at %ld\n", + thing->id, expected, index); + return -1; + } + expected += keep; + } + + /* + * Leave caches sparsely allocated. Shrink caches manually with: + * + * slabinfo radix_tree_node --shrink + * slabinfo smox_test --shrink + */ + + return 0; +} + +static int __init smox_cache_init(void) +{ + cachep = kmem_cache_create(SMOX_CACHE_NAME, + sizeof(struct smox_thing), + 0, 0, smox_object_ctor); + if (!cachep) + return -1; + + return 0; +} + +static void __exit smox_cache_cleanup(void) +{ + struct smox_thing *thing; + unsigned long i; + + xa_for_each(&things, i, thing) { + kmem_cache_free(cachep, thing); + } + xa_destroy(&things); + kmem_cache_destroy(cachep); +} + +static int __init smox_init(void) +{ + int ret; + + ret = smox_cache_init(); + if (ret) { + pr_err("smo_xarray: failed to create cache\n"); + return ret; + } + pr_info("smo_xarray: created kmem_cache: %s\n", SMOX_CACHE_NAME); + + kmem_cache_setup_mobility(cachep, NULL, smox_cache_migrate); + pr_info("smo_xarray: kmem_cache %s defrag enabled\n", SMOX_CACHE_NAME); + + /* + * Running this test consumes memory unless you shrink the + * radix_tree_node cache manually with `slabinfo`. + */ + ret = test_smo_xarray(); + if (ret) + pr_warn("test_smo_xarray failed: %d\n", ret); + + pr_info("smo_xarray: module loaded successfully\n"); + return 0; +} +module_init(smox_init); + +static void __exit smox_exit(void) +{ + smox_cache_cleanup(); + + pr_info("smo_xarray: module removed\n"); +} +module_exit(smox_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Tobin C. Harding"); +MODULE_DESCRIPTION("SMO XArray test module."); From patchwork Mon Jun 3 04:26:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972237 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 451E514DB for ; Mon, 3 Jun 2019 04:29:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 34C4A28174 for ; Mon, 3 Jun 2019 04:29:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 289D728387; Mon, 3 Jun 2019 04:29:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A972828174 for ; Mon, 3 Jun 2019 04:29:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A234C6B026E; Mon, 3 Jun 2019 00:29:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9D3F56B0276; Mon, 3 Jun 2019 00:29:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C3E36B0277; Mon, 3 Jun 2019 00:29:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id 6E56D6B026E for ; Mon, 3 Jun 2019 00:29:03 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id h198so13784781qke.1 for ; Sun, 02 Jun 2019 21:29:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ewjgpKzrSyNZxS5aNItsJCiFiD2mCvE5h0lTeGrIDUo=; b=CT6oh2n5Mtz0YKGx5T1nhXeBE/aPVOnmTcc2lfH/S93JISjl8dUdUC+4cayQcl1uXc cc+lb9qs5iLGWxAnaSBrlsOfvzWIN/VBGd6vWofsVL+diUHRR17fRnNs1zHLJAWeocAc AsGjl1hxTavXgtWVauKrVdi8GZ1vUbbaax47rqy0nxbnxQ6mxW7Iq1964r/pdJYi2Emg RagYyhEQ+ML2PJN8h+5QjgIk7Te3vROGAUejls4qk25rpxxBHKXwdOvhxhSZW9AWB3d8 ETIdqiCH3FObwrYuAEuN4lEM4TsrNJevkUS6N8IE3xa4ABbQ3NOpou8OCBIiYs2DH44/ 50CA== X-Gm-Message-State: APjAAAVqpVyghYrpvrYxbTBevIoCE7AkPkfYFQZf435XMwgqN+qUglP8 M9UhruU9C442Wk7ddEfEKGJwFc0UVRVfX+jAYsRSRRWlMqKaHhyG2pMY5TwkCd4Pj/TZA9Olxmc umgfZ0wdUEXmhdEg+EiDqjSBXW15tnMRd/rNQeAjAGtjCauBXzGI5HUuAZtsn/s0= X-Received: by 2002:aed:3145:: with SMTP id 63mr21077381qtg.184.1559536143226; Sun, 02 Jun 2019 21:29:03 -0700 (PDT) X-Google-Smtp-Source: APXvYqzLXV+YZ140LvJuhSyOLQckLxqosJEiPybOM32rNv2PERJxdhOa4RQtMARvndnARVj6CLEZ X-Received: by 2002:aed:3145:: with SMTP id 63mr21077343qtg.184.1559536142267; Sun, 02 Jun 2019 21:29:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536142; cv=none; d=google.com; s=arc-20160816; b=WAm7yTIVA3qe1Uv6fx8zkKBktCAFOrT6ZxcV4THULwUJoz4bqed0WCsmInYyYND1IT +AleWDBffOEjJRulIXu+1jPxOSgMEIMcIF3RqBP2JT3pu0Ip7XST+T2ALSA6rxWJ4lgc qlUlYglgtksO4+yTTkwO1uve5wChkGDV5A2QDPFUZMWulMZN5Zgc7k/bIo9CCKzLqlgo eY7bmbOj3sRPLfcaWkBlrncwi1WWvzJkJ4nRKMPM1wubhU2NQsHoUgkfr12pNaYoYhg7 tz0HCQbjUqxx/u/kzvNA+VXkdC2jQsyo0qDQZ8kBmmJN//vgGZ2RYXR4al3OdgUgNsQA WoOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=ewjgpKzrSyNZxS5aNItsJCiFiD2mCvE5h0lTeGrIDUo=; b=ZPQ1nCaruIaMFq2eR6gW+4MMPVxuhesmPntBnfSXzeR73384khXXoxJM8I0DXhGmuS zGdWtusiQqlWlFGQreeofUp5z8P/FYNd0bmUMkza5nHVnUyXJhYTMqgyYwHuXbYuHYvT eSjbiY+PTCLKVSUPRoV5hskhMxGi46PiSazEnm7cEY4UWuDwqpXJcbTgA8wfbGFrpToC 2CGG8+QLhfQbItZoTTDBI4zwxAtsRzmVpFJtCJvk9v1CRfLibrtN+gLqGItNvsXlNvG6 s2m9SF9rIO5+kzFUNbAaa6uyMvpdlvxbLAEIosG7K3E/n0kdgqp81/0iXK65ecyqvuFX 1BOA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=IoSRBbnW; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id o22si4594605qve.85.2019.06.02.21.29.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:29:02 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=IoSRBbnW; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 07BB11283; Mon, 3 Jun 2019 00:29:02 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:29:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=ewjgpKzrSyNZxS5aNItsJCiFiD2mCvE5h0lTeGrIDUo=; b=IoSRBbnW /MnkcmJqrNT5voVVHOF2HDm78r1ptlGnuLEAhtfByiYSAth7lBKRIFJHKGbONqmn n9UVthqyetkgCvsrj+7pkvH5MtmmzIiVs/c6jiLK7RxL/bbJUimKk3+O3szcTSRs vJIDHxqClhud2LL51vDDgg1FxOW68cw7tuSVnPX1iNMx24Ty5KeGH0+yAf0WrDBj socy5DCMQ7OE/FlOncvhs//KTN3ZJ5D1eoMOa9UGvVTkWNrQQx0IVoJ7tE8LCIud otq3mwQLk3BSy21ZvUTVfHv46iDriY2AepKcuL+2jT2Yu5mWCqSoEc4NY5EEUZyS IiYkQRy0xjTc3g== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id DCC0B80066; Mon, 3 Jun 2019 00:28:54 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 12/15] dcache: Provide a dentry constructor Date: Mon, 3 Jun 2019 14:26:34 +1000 Message-Id: <20190603042637.2018-13-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In order to support object migration on the dentry cache we need to have a determined object state at all times. Without a constructor the object would have a random state after allocation. Provide a dentry constructor. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index c435398f2c81..867d97a86940 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -1603,6 +1603,16 @@ void d_invalidate(struct dentry *dentry) } EXPORT_SYMBOL(d_invalidate); +static void dcache_ctor(void *p) +{ + struct dentry *dentry = p; + + /* Mimic lockref_mark_dead() */ + dentry->d_lockref.count = -128; + + spin_lock_init(&dentry->d_lock); +} + /** * __d_alloc - allocate a dcache entry * @sb: filesystem it will belong to @@ -1658,7 +1668,6 @@ struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) dentry->d_lockref.count = 1; dentry->d_flags = 0; - spin_lock_init(&dentry->d_lock); seqcount_init(&dentry->d_seq); dentry->d_inode = NULL; dentry->d_parent = dentry; @@ -3096,14 +3105,17 @@ static void __init dcache_init_early(void) static void __init dcache_init(void) { - /* - * A constructor could be added for stable state like the lists, - * but it is probably not worth it because of the cache nature - * of the dcache. - */ - dentry_cache = KMEM_CACHE_USERCOPY(dentry, - SLAB_RECLAIM_ACCOUNT|SLAB_PANIC|SLAB_MEM_SPREAD|SLAB_ACCOUNT, - d_iname); + slab_flags_t flags = + SLAB_RECLAIM_ACCOUNT | SLAB_PANIC | SLAB_MEM_SPREAD | SLAB_ACCOUNT; + + dentry_cache = + kmem_cache_create_usercopy("dentry", + sizeof(struct dentry), + __alignof__(struct dentry), + flags, + offsetof(struct dentry, d_iname), + sizeof_field(struct dentry, d_iname), + dcache_ctor); /* Hash may have been set up in dcache_init_early */ if (!hashdist) From patchwork Mon Jun 3 04:26:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972241 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A24381515 for ; Mon, 3 Jun 2019 04:29:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8DE5D28174 for ; Mon, 3 Jun 2019 04:29:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7F65528387; Mon, 3 Jun 2019 04:29:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DDDC128174 for ; Mon, 3 Jun 2019 04:29:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC0736B0277; Mon, 3 Jun 2019 00:29:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A70866B0278; Mon, 3 Jun 2019 00:29:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95F756B0279; Mon, 3 Jun 2019 00:29:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 766F76B0277 for ; Mon, 3 Jun 2019 00:29:10 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id n77so13708989qke.17 for ; Sun, 02 Jun 2019 21:29:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=xAUe/GSEE+822SPVzryZpZO8XwI9dFOU44eaj5D2zRU=; b=uFSYX8PIkBWs1Pwg6uRLB/91VWbJvC9QqqSOQQih+/bx5VvG9a9Nwm14r2+FepA84v leTFLbPRgbykO3Hl9hJdamtyqJ9I4sSvlxLxUGitiBxu9oi/+RevcThYAlMjGJZLpHA7 t9Yyo/KkWgCWqRFh+Pn8EGmcesT0CwbL2wwMrspnxBMMEi23ppxOL4iLmGvNCXJ+X0v3 0wYlap6qwURrbyGq+tNYymlTvjKg0ci/RcQBqDPZT40f86M0vJWxhoRcoGK4PwniI0Zd Ucrn2DSIicBw8DVXmzIMBj6z6UyDYl67JUyE8L4QRMIPpoejxIJ+MtStJG7y4pydAELB amUw== X-Gm-Message-State: APjAAAVGaGE7JqIOkr9s3h6hRFdCV1cL4uTsuovhlcsup/V0Bb0y0mhJ OFf30QNX0WVYWOx7ZrhEWcgeiSb/yAQkd67ik6lyo45CjwyANM2F2DetC6kPpI5fau16JxhFdKE dSMNRnE7wLhMUY90LaAWe3wwXFeJQZnEqpQ0pe09HCvDCemSKpn8mwaYuZUlWAGc= X-Received: by 2002:ac8:38d5:: with SMTP id g21mr20836485qtc.52.1559536150227; Sun, 02 Jun 2019 21:29:10 -0700 (PDT) X-Google-Smtp-Source: APXvYqy1hlKAKt3AJTKpd+R42+UNQmMRI0DEmaaEqCBEEXvSARuEWeMyx/rabRcVksLOnpEM1ez4 X-Received: by 2002:ac8:38d5:: with SMTP id g21mr20836453qtc.52.1559536149452; Sun, 02 Jun 2019 21:29:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536149; cv=none; d=google.com; s=arc-20160816; b=H32xkgGbashYABRkCN3NlruEF/OQiVCz0oIeVecCPxonmNVO3uilOJvTyZQG6Khw4G sp7y3jSjEF76Xks9PuY1YMOPQrbCrfeUYuHiCYlXFlEHe20UM6TapUICaWimDPhh3Y40 GzQ97LJf/UvEjLs3x3U/02QC71TBCBs5wPRbLiq+roS9lBfdR8nj5l1tgx2ubpiw661F lcxLv/cIGlciK3BF8Z6zbyCDdusbmOk1Ns4YeNc8+5kaDzK+U2/YVHV0xAS9zAeBxGm/ Z+jGu7glI9xi2Y9H8fTIH1P2/SlzGfF4PdJZ1RtCUt0IyL4YL51FZ7CjrHivSkOxrlzI v/MA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=xAUe/GSEE+822SPVzryZpZO8XwI9dFOU44eaj5D2zRU=; b=yGRiaOhNnErpEJo3NYtlAvduXguHLXuKOee0sXmaoXYoXHDj/ADqMnS5AuNH5jKzn3 VVo0VvphVn5Ih9pHT9A3+GuzfiyCALyqR6SlXfe9hn0zP7NWbtxkBonoeeaW6a2t27Wr Ka/R0JBC3wv8VYGemtaB1sBlITC3Oy5FUCV3aM1asmSlyX1GIFPqZRXaeInpcJMcL9KB 80j+aye2Gn4gXd4gnyaF0YhnFBgwxNAWsuCHJxT5rI5R9YjHsD8YpiV57840DPfEzhYG icpokxNNu1OBUnjN6YAIIUgPl7a7QltLbHxSdlMgw71TKkRtMz31yqvIAqxznx/S3MEu c2Xw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=imTMfnlo; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id j7si2414563qth.112.2019.06.02.21.29.09 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:29:09 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=imTMfnlo; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 3641A21A9; Mon, 3 Jun 2019 00:29:09 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:29:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=xAUe/GSEE+822SPVzryZpZO8XwI9dFOU44eaj5D2zRU=; b=imTMfnlo DxJpQ3ZxqMsQ2FeM4DiG4WgLXKr5msfS1/U/H5OrxPpDNaXEwnxHt8wwMp8oC7L7 qSZAJSfFctSrnVTckBMZGqjeXdCrcd1oHGlX9em+unJm5R3wii3FXU3s6j/SO/s1 JbnEVn2Z4DyDP2bkXpiBaXBH1zzCfH0QBVQy4kVden0T4LwoZH2AuV7FosgiURg7 vsNs0fJPAwfCl4AEl9frWtjJrqBIglSha7eFWUfteIrLpTorvhhugU8jANxdOovL GK8/mzteXjES6m4AM50S/0nZUhcdjo1fiIc3EVKknK5JJmkbOV5YZDa878SXgF4W 0AWGw9QQxSU24A== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id 1FD568005C; Mon, 3 Jun 2019 00:29:01 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 13/15] dcache: Implement partial shrink via Slab Movable Objects Date: Mon, 3 Jun 2019 14:26:35 +1000 Message-Id: <20190603042637.2018-14-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The dentry slab cache is susceptible to internal fragmentation. Now that we have Slab Movable Objects we can attempt to defragment the dcache. Dentry objects are inherently _not_ relocatable however under some conditions they can be free'd. This is the same as shrinking the dcache but instead of shrinking the whole cache we only attempt to free those objects that are located in partially full slab pages. There is no guarantee that this will reduce the memory usage of the system, it is a compromise between fragmented memory and total cache shrinkage with the hope that some memory pressure can be alleviated. This is implemented using the newly added Slab Movable Objects infrastructure. The dcache 'migration' function is intentionally _not_ called 'd_migrate' because we only free, we do not migrate. Call it 'd_partial_shrink' to make explicit that no reallocation is done. In order to enable SMO a call to kmem_cache_setup_mobility() must be made, we do this during initialization of the dcache. Implement isolate and 'migrate' functions for the dentry slab cache. Enable SMO for the dcache during initialization. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) diff --git a/fs/dcache.c b/fs/dcache.c index 867d97a86940..3ca721752723 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -3072,6 +3072,79 @@ void d_tmpfile(struct dentry *dentry, struct inode *inode) } EXPORT_SYMBOL(d_tmpfile); +/* + * d_isolate() - Dentry isolation callback function. + * @s: The dentry cache. + * @v: Vector of pointers to the objects to isolate. + * @nr: Number of objects in @v. + * + * The slab allocator is holding off frees. We can safely examine + * the object without the danger of it vanishing from under us. + */ +static void *d_isolate(struct kmem_cache *s, void **v, int nr) +{ + struct list_head *dispose; + struct dentry *dentry; + int i; + + dispose = kmalloc(sizeof(*dispose), GFP_KERNEL); + if (!dispose) + return NULL; + + INIT_LIST_HEAD(dispose); + + for (i = 0; i < nr; i++) { + dentry = v[i]; + spin_lock(&dentry->d_lock); + + if (dentry->d_lockref.count > 0 || + dentry->d_flags & DCACHE_SHRINK_LIST) { + spin_unlock(&dentry->d_lock); + continue; + } + + if (dentry->d_flags & DCACHE_LRU_LIST) + d_lru_del(dentry); + + d_shrink_add(dentry, dispose); + spin_unlock(&dentry->d_lock); + } + + return dispose; +} + +/* + * d_partial_shrink() - Dentry migration callback function. + * @s: The dentry cache. + * @_unused: We do not access the vector. + * @__unused: No need for length of vector. + * @___unused: We do not do any allocation. + * @private: list_head pointer representing the shrink list. + * + * Dispose of the shrink list created during isolation function. + * + * Dentry objects can _not_ be relocated and shrinking the whole dcache + * can be expensive. This is an effort to free dentry objects that are + * stopping slab pages from being free'd without clearing the whole dcache. + * + * This callback is called from the SLUB allocator object migration + * infrastructure in attempt to free up slab pages by freeing dentry + * objects from partially full slabs. + */ +static void d_partial_shrink(struct kmem_cache *s, void **_unused, int __unused, + int ___unused, void *private) +{ + struct list_head *dispose = private; + + if (!private) /* kmalloc error during isolate. */ + return; + + if (!list_empty(dispose)) + shrink_dentry_list(dispose); + + kfree(private); +} + static __initdata unsigned long dhash_entries; static int __init set_dhash_entries(char *str) { @@ -3117,6 +3190,8 @@ static void __init dcache_init(void) sizeof_field(struct dentry, d_iname), dcache_ctor); + kmem_cache_setup_mobility(dentry_cache, d_isolate, d_partial_shrink); + /* Hash may have been set up in dcache_init_early */ if (!hashdist) return; From patchwork Mon Jun 3 04:26:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972245 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 50B001515 for ; Mon, 3 Jun 2019 04:29:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3FC2728174 for ; Mon, 3 Jun 2019 04:29:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 32C5F28387; Mon, 3 Jun 2019 04:29:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5602028174 for ; Mon, 3 Jun 2019 04:29:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3E1096B026F; Mon, 3 Jun 2019 00:29:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 392B96B0279; Mon, 3 Jun 2019 00:29:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 281516B027A; Mon, 3 Jun 2019 00:29:18 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 077566B026F for ; Mon, 3 Jun 2019 00:29:18 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id z8so6525989qti.1 for ; Sun, 02 Jun 2019 21:29:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=iA0DTRJv0E/AcBU5UOtkTZ65bco2wdgu9U89vbF4jbM=; b=SkC2yOerxPid4kse5XZZBLVTF4hhQs5LWfllJxyptFjnfV014QRNcfd3qOqLVSIRGf 9mnzOEL6dymXC7hHQ4kdCTB5y48hoC28DLarMBP3H1G6MotAE10IfONJSDZgSyQNsCCu 0ipcYgp69xf5xXt/iSePuA/aoGpdwI3YMIRTurDGfQa0vnIyBF7qaDQRdKbOHCZFK7JP 0Wi8v4KuoKpI6lTwPkIh6pIQJoHjithfOPx2ep3wJHwx5ueT953fhtcHUuFgBThhUD4S vqm2xfTGWdAZkhvaIrrzuXnhfKww/dXaIVFr0rbwmt0p0RLUMFlZSqBeK5M0BToPSYO3 sk3A== X-Gm-Message-State: APjAAAW7fqV3CdMssOM/bmHVWyGseX/wCNPQVqhRTJoytKV48l7rgTp0 g86qFDH6BAlD6pYtjIo8ry3nO91grjQt/np3i6mfM+ziNtIHL57cK7WuUyA8kbDf1lTB6h7+725 un4ehYnRtxEEshqi6Ufm2uKMu3sD3SQ1Afe3hK8X9+ctTpqhVpC/kitISDrIx6nA= X-Received: by 2002:a0c:afa2:: with SMTP id s31mr20047855qvc.186.1559536157750; Sun, 02 Jun 2019 21:29:17 -0700 (PDT) X-Google-Smtp-Source: APXvYqzKLSQyOtnj7huF8FXj9gICDzp44kFZFGfXq+NWxrqg4wiyL6iiUxoBkmkLLRLdbnoslFqe X-Received: by 2002:a0c:afa2:: with SMTP id s31mr20047826qvc.186.1559536156723; Sun, 02 Jun 2019 21:29:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536156; cv=none; d=google.com; s=arc-20160816; b=O+7fX877BoD7h7B7WbL/wKM5CGA3OKKhfrpv3sa6Xb6llrJ+ckcXKKEt0CRoGrkt6N B2U6LkLCAMvORw3FcfdOYAPiPFytsitQF2sFGeYZ6rGJMmLup1rkvOHh4nDRjMyeymCF 149s/1QdTpAJuoaldDdweoit/WHa/2KQFx6su8D9b9a5VpgceVt4i430EKQnirw2alS8 yyomk6+PWEJW8uuSvjVmHMaEn2pcLqUGLe9hvJ7cSb1XDZa7jw7cxL4iSzbs9tWFkzMR TBGqrlWfdNFI0tGNW/ARE54E2uJUTRp+eL9SyR5yBclZgn2pn/9A+VzzFBqgg66ZWYVF Tirw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=iA0DTRJv0E/AcBU5UOtkTZ65bco2wdgu9U89vbF4jbM=; b=WHn/02GIMXfUAEx5VOBFywaoWakO6Z+xdcHE9aba72xBTgEIH4ecH4L5zzXuHnTYgs pWPbHkO7d/tyUjm0jT5vFup1Otc66mdKoESxvKaH4O+ce64Ey62ppMO43BctjyjGM01H IcL/oDU66xih/8pQrQVd36vRRb7dCZUA9BVYXmjrGUWa6wZu6Be2s4EnGfax187GgHuT ZFH8L9noQEPWKlDMlbbO4cKKImOWQX3NyZRXtBLwcJ/PaX8GDIw5q/sn+3tOxtf36QXW LHP7It/J/XY8k7Bk2qYwmq6l1RHMtcaQ/jC2qjb2H5al1128Ln279KJVNNiG/G22NO+o NcTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ipriwsol; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id v1si1976770qtk.325.2019.06.02.21.29.16 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:29:16 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ipriwsol; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 7635321E9; Mon, 3 Jun 2019 00:29:16 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:29:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=iA0DTRJv0E/AcBU5UOtkTZ65bco2wdgu9U89vbF4jbM=; b=ipriwsol +7UaVqrLIu1u53AstwTXvya8XvHS7t6yyPocTiqNiP/neGGSbovHXfWvBP/Ksh3v 5ZLSj1p/PnBzapI5nSK65KrE42OmeBWqZFX9mnJfAsfvCEsdpWKPcQctBoZYm0SC EZiBBry9i3VOfgEUHUNJgDbkNVA7/P6ahMdJMdikn2uTV+m+OE4IM3LaHTFazeSP 2e8UU4WzH0umSWDyitb+G2knCRNR8/R0oi2FLx+Hxq/w8CKD8ZqHUDn9jHNwNvlw dplFud7AaD7o1hIZYa4L71fe+IMJDNPDgQl7i7nIS2xkf3GyOL1vXJqrO0sAHaez IFVpCBpA++7piQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id 54EB78005C; Mon, 3 Jun 2019 00:29:09 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 14/15] slub: Enable moving objects to/from specific nodes Date: Mon, 3 Jun 2019 14:26:36 +1000 Message-Id: <20190603042637.2018-15-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We have just implemented Slab Movable Objects (SMO, object migration). Currently object migration is used to defrag a cache. On NUMA systems it would be nice to be able to control the source and destination nodes when moving objects. Add CONFIG_SLUB_SMO_NODE to guard this feature. CONFIG_SLUB_SMO_NODE depends on CONFIG_SLUB_DEBUG because we use the full list. Implement moving all objects (including those in full slabs) to a specific node. Expose this functionality to userspace via a sysfs entry. Add sysfs entry: /sysfs/kernel/slab//move With this users get access to the following functionality: - Move all objects to specified node. echo "N1" > move - Move all objects from specified node to other specified node (from N1 -> to N2): echo "N1 N2" > move This also enables shrinking slabs on a specific node: echo "N1 N1" > move Signed-off-by: Tobin C. Harding --- mm/Kconfig | 7 ++ mm/slub.c | 247 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 254 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index f0c76ba47695..c1438b9e578b 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -259,6 +259,13 @@ config ARCH_ENABLE_THP_MIGRATION config CONTIG_ALLOC def_bool (MEMORY_ISOLATION && COMPACTION) || CMA +config SLUB_SMO_NODE + bool "Enable per node control of Slab Movable Objects" + depends on SLUB && SYSFS + select SLUB_DEBUG + help + On NUMA systems enable moving objects to and from a specified node. + config PHYS_ADDR_T_64BIT def_bool 64BIT diff --git a/mm/slub.c b/mm/slub.c index 2157205df7ba..23566e5a712b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4336,6 +4336,130 @@ static void move_slab_page(struct page *page, void *scratch, int node) s->migrate(s, vector, count, node, private); } +#ifdef CONFIG_SLUB_SMO_NODE +/* + * kmem_cache_move() - Attempt to move all slab objects. + * @s: The cache we are working on. + * @node: The node to move objects away from. + * @target_node: The node to move objects on to. + * + * Attempts to move all objects (partial slabs and full slabs) to target + * node. + * + * Context: Takes the list_lock. + * Return: The number of slabs remaining on node. + */ +static unsigned long kmem_cache_move(struct kmem_cache *s, + int node, int target_node) +{ + struct kmem_cache_node *n = get_node(s, node); + LIST_HEAD(move_list); + struct page *page, *page2; + unsigned long flags; + void **scratch; + + if (!s->migrate) { + pr_warn("%s SMO not enabled, cannot move objects\n", s->name); + goto out; + } + + scratch = alloc_scratch(s); + if (!scratch) + goto out; + + spin_lock_irqsave(&n->list_lock, flags); + + list_for_each_entry_safe(page, page2, &n->partial, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + if (page->inuse) { + list_move(&page->lru, &move_list); + /* Stop page being considered for allocations */ + n->nr_partial--; + page->frozen = 1; + + slab_unlock(page); + } else { /* Empty slab page */ + list_del(&page->lru); + n->nr_partial--; + slab_unlock(page); + discard_slab(s, page); + } + } + list_for_each_entry_safe(page, page2, &n->full, lru) { + if (!slab_trylock(page)) + continue; + + list_move(&page->lru, &move_list); + page->frozen = 1; + slab_unlock(page); + } + + spin_unlock_irqrestore(&n->list_lock, flags); + + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + + /* Bail here to save taking the list_lock */ + if (list_empty(&move_list)) + goto out; + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + if (page->inuse == page->objects) { + list_add(&page->lru, &n->full); + slab_unlock(page); + } else { + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); +out: + return atomic_long_read(&n->nr_slabs); +} + +/* + * kmem_cache_move_to_node() - Move all slab objects to node. + * @s: The cache we are working on. + * @node: The target node to move objects to. + * + * Attempt to move all slab objects from all nodes to @node. + * + * Return: The total number of slabs left on emptied nodes. + */ +static unsigned long kmem_cache_move_to_node(struct kmem_cache *s, int node) +{ + unsigned long left = 0; + int nid; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == node) + continue; + + left += kmem_cache_move(s, nid, node); + } + + return left; +} +#endif /* CONFIG_SLUB_SMO_NODE */ + /* * kmem_cache_defrag() - Defragment node. * @s: cache we are working on. @@ -5594,6 +5718,126 @@ static ssize_t shrink_store(struct kmem_cache *s, } SLAB_ATTR(shrink); +#ifdef CONFIG_SLUB_SMO_NODE +static ssize_t move_show(struct kmem_cache *s, char *buf) +{ + return 0; +} + +/* + * parse_move_store_input() - Parse buf getting integer arguments. + * @buf: Buffer to parse. + * @length: Length of @buf. + * @arg0: Return parameter, first argument. + * @arg1: Return parameter, second argument. + * + * Parses the input from user write to sysfs file 'move'. Input string + * should contain either one or two node specifiers of form Nx where x + * is an integer specifying the NUMA node ID. 'N' or 'n' may be used. + * n/N may be omitted. + * + * e.g. + * echo 'N1' > /sysfs/kernel/slab/cache/move + * or + * echo 'N0 N2' > /sysfs/kernel/slab/cache/move + * + * Regex matching accepted forms: '[nN]?[0-9]( [nN]?[0-9])?' + * + * FIXME: This is really fragile. Input must be exactly correct, + * spurious whitespace causes parse errors. + * + * Return: 0 if an argument was successfully converted, or an error code. + */ +static ssize_t parse_move_store_input(const char *buf, size_t length, + long *arg0, long *arg1) +{ + char *s, *save, *ptr; + int ret = 0; + + if (!buf) + return -EINVAL; + + s = kstrdup(buf, GFP_KERNEL); + if (!s) + return -ENOMEM; + save = s; + + if (s[length - 1] == '\n') { + s[length - 1] = '\0'; + length--; + } + + ptr = strsep(&s, " "); + if (!ptr || strcmp(ptr, "") == 0) { + ret = 0; + goto out; + } + + if (*ptr == 'N' || *ptr == 'n') + ptr++; + ret = kstrtol(ptr, 10, arg0); + if (ret < 0) + goto out; + + if (s) { + if (*s == 'N' || *s == 'n') + s++; + ret = kstrtol(s, 10, arg1); + if (ret < 0) + goto out; + } + + ret = 0; +out: + kfree(save); + return ret; +} + +static bool is_valid_node(int node) +{ + int nid; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == node) + return true; + } + return false; +} + +/* + * move_store() - Move objects between nodes. + * @s: The cache we are working on. + * @buf: String received. + * @length: Length of @buf. + * + * Writes to /sys/kernel/slab//move are interpreted as follows: + * + * echo "N1" > move : Move all objects (from all nodes) to node 1. + * echo "N0 N1" > move : Move all objects from node 0 to node 1. + * + * 'N' may be omitted: + */ +static ssize_t move_store(struct kmem_cache *s, const char *buf, size_t length) +{ + long arg0 = -1; + long arg1 = -1; + int ret; + + ret = parse_move_store_input(buf, length, &arg0, &arg1); + if (ret < 0) + return -EINVAL; + + if (is_valid_node(arg0) && is_valid_node(arg1)) + (void)kmem_cache_move(s, arg0, arg1); + else if (is_valid_node(arg0)) + (void)kmem_cache_move_to_node(s, arg0); + + /* FIXME: What should we be returning here? */ + return length; +} +SLAB_ATTR(move); +#endif /* CONFIG_SLUB_SMO_NODE */ + #ifdef CONFIG_NUMA static ssize_t remote_node_defrag_ratio_show(struct kmem_cache *s, char *buf) { @@ -5718,6 +5962,9 @@ static struct attribute *slab_attrs[] = { &reclaim_account_attr.attr, &destroy_by_rcu_attr.attr, &shrink_attr.attr, +#ifdef CONFIG_SLUB_SMO_NODE + &move_attr.attr, +#endif &slabs_cpu_partial_attr.attr, #ifdef CONFIG_SLUB_DEBUG &total_objects_attr.attr, From patchwork Mon Jun 3 04:26:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10972249 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6708914DB for ; Mon, 3 Jun 2019 04:29:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 55C1E28174 for ; Mon, 3 Jun 2019 04:29:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4950C28387; Mon, 3 Jun 2019 04:29:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 954D928174 for ; Mon, 3 Jun 2019 04:29:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F2096B0279; Mon, 3 Jun 2019 00:29:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4A3DD6B027A; Mon, 3 Jun 2019 00:29:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36A8F6B027B; Mon, 3 Jun 2019 00:29:25 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 1547F6B0279 for ; Mon, 3 Jun 2019 00:29:25 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id l11so3414590qtp.22 for ; Sun, 02 Jun 2019 21:29:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=u3v0eiZffY+v2sn8V9SnXJ0gtCWl1JCgJq73oZGa8dI=; b=FceihokTILCqS8kH70aDI+iFr2V0COHl+JEbF4RmN3t3k6ElP+yPV4li8dLZ7J3mqT VEFp7n1dl3ggQVtB2Cb7OaeK4S5sas7Tz2HM/hS/U1o4QCUMWyNAIfdGs5VESS+htQZE 8VAynXs0xq0QP/kFgH5rVVtnt0M+gPkV3JUrCG/j/AQcu7WTkVNogpuu6j3JQx3g+HE2 UIA7o1YSBpTUSrh0G/T6kbrtSSNfERoyqFsXZDA44Cl77RLwtam3q96Q4DE9vOOGDAGz 5I8+Y/u0o/Y2w3gHCZqer/wkkCYkdXKVNt67977JhlCTmZzdZ3zOsNc5wBFv/gO/eCf8 J4lA== X-Gm-Message-State: APjAAAW+JQnx+Q9Q+BUwXR42+47ocmsPFsTscD1oFdvG+GqhrXig3jr2 4ZbVEc5zW62hHSy2lWn+r3punuvqvgSu+lO7v57KTaA2tqjNKerxrKVMt5NbpfGmooX2NPrOJiY 0gNLBDt/PDs7Z5PFi7jSJoSXnfrPW4Aq+27EuByBujrVLaAwue6dBLbdsAsMHN1M= X-Received: by 2002:ac8:6984:: with SMTP id o4mr21012200qtq.122.1559536164818; Sun, 02 Jun 2019 21:29:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqwWtW5A6aIojwDR+Ct0s1cjEsuFB0mmAgkjbTNFn4Kc+DxxOKwlZWspmEZAFUAIUNSeYHbU X-Received: by 2002:ac8:6984:: with SMTP id o4mr21012170qtq.122.1559536163998; Sun, 02 Jun 2019 21:29:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559536163; cv=none; d=google.com; s=arc-20160816; b=S5v9GTz9/G5CiFsG9P1jM3F5zDQzKrpEcJdbZU6X/44H8G1zgjml9hranQaCrYtGGn RCVh1ncZZ9NE6tuYQ0VzHpHH1+f092oSY/pXYETbfudOW1KsXic9r0DT4NWtplWpfwGQ NUGyuglCvmVKJdquhrc8vIVPiXf1uJRPm+9vCblP2nXrwOiA4vabg1f89VuuljKT/YU/ 9CVecImO2tbCcLuVJ59Hpe9FN/vcy+7vo44Pk+jnqM/F4RpLeE+Km2YApN/SZbqEsEza 1FskFpLzAYwst1X7tpEUlXxSBcXHOo3OvfU1aAqds8aFwLra3UlQJMvp5zJi3BURUgIA 4ejw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=u3v0eiZffY+v2sn8V9SnXJ0gtCWl1JCgJq73oZGa8dI=; b=cmhhfzIrh9tsNY2E3D9lqYp77qpBjlxrQbHYCR2SbsmeAbH6ZNIQ+zwrjAwstcaheQ czAc85iVM9yHY6Sdx+xrH97vXmZ0rG7okMGWY/+Axhdz50RhYnfn/bEKBoKVYee0U4B6 U5bKm5in89qBZNr5z8PYaDjXfXtCvbsqIyJx4iE0JfMrZDe8CNFYthYn/9I0gNU4afDc SdRGEaOC5mdprIFypbYboz/C4XdBpFUdvo2rGloUvI2Zq9xuU2zzG8rR9b0KXKAopfTE EgUghwg0+GUFCLQAM9fjCvDMXXnC2LPaLfQsoFmzDB48yCg/tSdKwnpVynMXbz5DxT/a LdmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=LPQlwJrO; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id c8si9144229qkl.265.2019.06.02.21.29.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 02 Jun 2019 21:29:23 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=LPQlwJrO; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id BF63821FB; Mon, 3 Jun 2019 00:29:23 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Jun 2019 00:29:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=u3v0eiZffY+v2sn8V9SnXJ0gtCWl1JCgJq73oZGa8dI=; b=LPQlwJrO rlkK/aTKLE3ZbR1UfmcfzlAUCyKPw8waLGQpzkYI/A5+0IKEnsjJYFiMUoD8MDkj vrHDs2Ya0zR7ue81LL1ThhvGv8BimaSnUO9o6q1Lwt09kbSu7ZxchL1OQteUlPRR 7TuZ+oS/qf3nJ2oT0uqWSGSaPJ0UWQX4UGcbRHdiRq32EQzkgCpOOYZDWaBxbDAm 2oIWR+76Jkwz5DjIXVBUpxmSEWbNUMo35tOH7i4WBJgkm4F0MSBJBY9jmNgjC6g4 4zsmGirt8UPn1jynaz2Eei9l2/1GU3t43bhb1qgvnFXaPBS5rbUm90UCcp1+u6CA uhWce+zErqC7jQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudefiedgkedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpedfvfhosghi nhcuvedrucfjrghrughinhhgfdcuoehtohgsihhnsehkvghrnhgvlhdrohhrgheqnecukf hppeduvdegrddugeelrdduudefrdefieenucfrrghrrghmpehmrghilhhfrhhomhepthho sghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-149-113-36.dyn.iinet.net.au [124.149.113.36]) by mail.messagingengine.com (Postfix) with ESMTPA id 9CF3C8005B; Mon, 3 Jun 2019 00:29:16 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , Matthew Wilcox , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 15/15] slub: Enable balancing slabs across nodes Date: Mon, 3 Jun 2019 14:26:37 +1000 Message-Id: <20190603042637.2018-16-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190603042637.2018-1-tobin@kernel.org> References: <20190603042637.2018-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We have just implemented Slab Movable Objects (SMO). On NUMA systems slabs can become unbalanced i.e. many slabs on one node while other nodes have few slabs. Using SMO we can balance the slabs across all the nodes. The algorithm used is as follows: 1. Move all objects to node 0 (this has the effect of defragmenting the cache). 2. Calculate the desired number of slabs for each node (this is done using the approximation nr_slabs / nr_nodes). 3. Loop over the nodes moving the desired number of slabs from node 0 to the node. Feature is conditionally built in with CONFIG_SMO_NODE, this is because we need the full list (we enable SLUB_DEBUG to get this). Future version may separate final list out of SLUB_DEBUG. Expose this functionality to userspace via a sysfs entry. Add sysfs entry: /sysfs/kernel/slab//balance Write of '1' to this file triggers balance, no other value accepted. This feature relies on SMO being enable for the cache, this is done with a call to, after the isolate/migrate functions have been defined. kmem_cache_setup_mobility(s, isolate, migrate) Signed-off-by: Tobin C. Harding --- mm/slub.c | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 23566e5a712b..70e46c4db757 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4458,6 +4458,119 @@ static unsigned long kmem_cache_move_to_node(struct kmem_cache *s, int node) return left; } + +/* + * kmem_cache_move_slabs() - Attempt to move @num slabs to target_node, + * @s: The cache we are working on. + * @node: The node to move objects from. + * @target_node: The node to move objects to. + * @num: The number of slabs to move. + * + * Attempts to move @num slabs from @node to @target_node. This is done + * by migrating objects from slabs on the full_list. + * + * Return: The number of slabs moved or error code. + */ +static long kmem_cache_move_slabs(struct kmem_cache *s, + int node, int target_node, long num) +{ + struct kmem_cache_node *n = get_node(s, node); + LIST_HEAD(move_list); + struct page *page, *page2; + unsigned long flags; + void **scratch; + long done = 0; + + if (!s->migrate) { + pr_warn("%s SMO not enabled, cannot move objects\n", s->name); + goto out; + } + + if (node == target_node) + return -EINVAL; + + scratch = alloc_scratch(s); + if (!scratch) + return -ENOMEM; + + spin_lock_irqsave(&n->list_lock, flags); + + list_for_each_entry_safe(page, page2, &n->full, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + list_move(&page->lru, &move_list); + page->frozen = 1; + slab_unlock(page); + + if (++done >= num) + break; + } + spin_unlock_irqrestore(&n->list_lock, flags); + + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + + /* Bail here to save taking the list_lock */ + if (list_empty(&move_list)) + goto out; + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + /* + * This is best effort only, if slab still has + * objects just put it back on the partial list. + */ + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); +out: + return done; +} + +/* + * kmem_cache_balance_nodes() - Balance slabs across nodes. + * @s: The cache we are working on. + */ +static void kmem_cache_balance_nodes(struct kmem_cache *s) +{ + struct kmem_cache_node *n = get_node(s, 0); + unsigned long desired_nr_slabs_per_node; + unsigned long nr_slabs; + int nr_nodes = 0; + int nid; + + (void)kmem_cache_move_to_node(s, 0); + + for_each_node_state(nid, N_NORMAL_MEMORY) + nr_nodes++; + + nr_slabs = atomic_long_read(&n->nr_slabs); + desired_nr_slabs_per_node = nr_slabs / nr_nodes; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == 0) + continue; + + kmem_cache_move_slabs(s, 0, nid, desired_nr_slabs_per_node); + } +} #endif /* CONFIG_SLUB_SMO_NODE */ /* @@ -5836,6 +5949,22 @@ static ssize_t move_store(struct kmem_cache *s, const char *buf, size_t length) return length; } SLAB_ATTR(move); + +static ssize_t balance_show(struct kmem_cache *s, char *buf) +{ + return 0; +} + +static ssize_t balance_store(struct kmem_cache *s, + const char *buf, size_t length) +{ + if (buf[0] == '1') + kmem_cache_balance_nodes(s); + else + return -EINVAL; + return length; +} +SLAB_ATTR(balance); #endif /* CONFIG_SLUB_SMO_NODE */ #ifdef CONFIG_NUMA @@ -5964,6 +6093,7 @@ static struct attribute *slab_attrs[] = { &shrink_attr.attr, #ifdef CONFIG_SLUB_SMO_NODE &move_attr.attr, + &balance_attr.attr, #endif &slabs_cpu_partial_attr.attr, #ifdef CONFIG_SLUB_DEBUG