From patchwork Mon May 20 05:40:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949889 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF49F17D2 for ; Mon, 20 May 2019 05:41:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9E4142022C for ; Mon, 20 May 2019 05:41:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 920E92863C; Mon, 20 May 2019 05:41:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2B5628635 for ; Mon, 20 May 2019 05:41:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B28156B0006; Mon, 20 May 2019 01:41:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id AD8796B0007; Mon, 20 May 2019 01:41:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A0766B0008; Mon, 20 May 2019 01:41:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 77CB56B0006 for ; Mon, 20 May 2019 01:41:19 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id b46so13231713qte.6 for ; Sun, 19 May 2019 22:41:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=/Io00D878lZ0GkrFvR4yMjAJgpoguE22uvDOPIw7E14=; b=b3JPw1l+fplU4XB+KWq0MAcCz2/KmfKYPdhn8JjWqSdRf9XCTaQyQY3GTNkOsrgbLF NO9fKSxnwlRM3/Kn/FUitvwm99Aoobh7+uaiSzKoRsOriebotN2M1uPULhXxd1kD+qca 1i/Tv0/rSVaR5s/Hx+ZIb0vnKmN+IFi9L/OAiJdBhYQcQEK/r/KhL2G53w6qtJoX6oUK AeXnZ81wO06S4HR3T+bn5DuwWvPTGvlRNrntUEOngrL91VWu0FKXMY2PENhuDxntDeMk 5mnjjChIARjB4+eH0Mu2YHgcOLu+vW/SHmjXOgpti+s/aUzXNgzSgM/ioG0/g2ddSgFv ROHg== X-Gm-Message-State: APjAAAWJFHNyoyiEZLF/UufDcq7U8nUDtfB/hWc05bYqm6kEU369wDar VU26SKcGQM7TFUnSJazTUdMX0xZFQbQbplvtnRYZ1L2yE1/vOAdHU3/DwVn4PcJcDg0zkabKH90 MKyWwhkSlA6NvjbKrs6s61gw/cvichAIRwFyaHjYWjaEWnmriXFr+tuftZEDXbfE= X-Received: by 2002:ac8:60d4:: with SMTP id i20mr54222886qtm.376.1558330879235; Sun, 19 May 2019 22:41:19 -0700 (PDT) X-Google-Smtp-Source: APXvYqwFfsfIlUpDyKgbxl4C/umH5SbsIfoOnSXNsZoYgO8HAfml9YiNH0/eaQcqA9wcTr/0CvEz X-Received: by 2002:ac8:60d4:: with SMTP id i20mr54222812qtm.376.1558330877783; Sun, 19 May 2019 22:41:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330877; cv=none; d=google.com; s=arc-20160816; b=YgLwU5OjIOoqSXJOg0JblmCVZYoLjv7dPDk16yWqS0fj9r+WkTZXSm5pmbBMYY8lEA Z6E2MLxxjcWoYiudToyXfEdVUbrjRivSXHPJsNRGeWuWsFd3CM3+LJzCinrlwdsus6GV m1qakL36FxQc9KESs5SaTx7O+cn3CBZFFOrt2VK9SlOVbjkvVXbFSYT+YPAYiQO23lLM lu9RIO1/fhVAlRruRO1uSsZKG8Ciyqyz1C/pBigsyptq+9LjTR/h8QPma2lDeLr+6RUQ Q7jjLBWIagsKdUTnlb5I97DrhvZLNswQCKlVMhEAdOTGeMKlxJCYIXz6ibIUGkmpBCiH GKxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=/Io00D878lZ0GkrFvR4yMjAJgpoguE22uvDOPIw7E14=; b=aDkgPYZ/NE5HdI3Iqv7Z316zuRuXRAslEhC/vjOx4lnUXO7WT0T7dQyMFhiqNQWzok OyrJanCDi9j69Wb5YroDT7mfUwdekLuPmIU8mbyDhkxcV+8phQNbN6ua29+sNOICxnVd baewnR/TbHXLlzuHiGsoPKT9ZLrps4rhqY2vcBCEjmWbyQ9658HLqXJvIWC1Otn6e1zb 2bxigwFdQ+pJgRkmnKtT+gdx/MCtD7oiHoM6IRt9HwXWCecBzHJbcQoB0AhWhX0rh8mg Az7klJNlPgFVrnebx6u4RLUk45THRTXvxqSBPpJqlJkkRTidtb9ygh4fW2wMjNwDopvf fXBg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=RReBq3uf; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id o185si8033101qkd.256.2019.05.19.22.41.17 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:41:17 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=RReBq3uf; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 7A3DBF012; Mon, 20 May 2019 01:41:17 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:41:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=/Io00D878lZ0GkrFvR4yMjAJgpoguE22uvDOPIw7E14=; b=RReBq3uf YVGO/bPlbhRNtkSgs5+ZAFxsCb7IKnF5f3Rf5w1KQvTyRNBb4iZV3/7EyRM/y4aF TIa6n2zBMITaGCDSCdkr0Lt57+834/eRjm84VBIi18qnpyj7nwlDUKKk/UUKSqj+ yJaJNkRi1v2TtIo2MlYkBebXQnms6oN2IzcgS4BR3zbYX4jE7nvTrKf1yuTlMes3 kq4kW81wSvZREvoGW4tUzpUTiNqBSxYCX/88L50/n40kwi0BqJ2nEMWXBj055zad Rmg7JmST1XKtFdx9hkiGo9Jqui2mQVHUgNInfs19FqQjdCgJUhm9bkNVSf4PcKfO l7uGUx9O0xuRiw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 4B96580062; Mon, 20 May 2019 01:41:10 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 01/16] slub: Add isolate() and migrate() methods Date: Mon, 20 May 2019 15:40:02 +1000 Message-Id: <20190520054017.32299-2-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the two methods needed for moving objects and enable the display of the callbacks via the /sys/kernel/slab interface. Add documentation explaining the use of these methods and the prototypes for slab.h. Add functions to setup the callbacks method for a slab cache. Add empty functions for SLAB/SLOB. The API is generic so it could be theoretically implemented for these allocators as well. Change sysfs 'ctor' field to be 'ops' to contain all the callback operations defined for a slab cache. Display the existing 'ctor' callback in the ops fields contents along with 'isolate' and 'migrate' callbacks. Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding Reviewed-by: Roman Gushchin --- include/linux/slab.h | 70 ++++++++++++++++++++++++++++++++++++++++ include/linux/slub_def.h | 3 ++ mm/slub.c | 59 +++++++++++++++++++++++++++++---- 3 files changed, 126 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..886fc130334d 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -154,6 +154,76 @@ void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); void memcg_destroy_kmem_caches(struct mem_cgroup *); +/* + * Function prototypes passed to kmem_cache_setup_mobility() to enable + * mobile objects and targeted reclaim in slab caches. + */ + +/** + * typedef kmem_cache_isolate_func - Object migration callback function. + * @s: The cache we are working on. + * @ptr: Pointer to an array of pointers to the objects to isolate. + * @nr: Number of objects in @ptr array. + * + * The purpose of kmem_cache_isolate_func() is to pin each object so that + * they cannot be freed until kmem_cache_migrate_func() has processed + * them. This may be accomplished by increasing the refcount or setting + * a flag. + * + * The object pointer array passed is also passed to + * kmem_cache_migrate_func(). The function may remove objects from the + * array by setting pointers to %NULL. This is useful if we can + * determine that an object is being freed because + * kmem_cache_isolate_func() was called when the subsystem was calling + * kmem_cache_free(). In that case it is not necessary to increase the + * refcount or specially mark the object because the release of the slab + * lock will lead to the immediate freeing of the object. + * + * Context: Called with locks held so that the slab objects cannot be + * freed. We are in an atomic context and no slab operations + * may be performed. + * Return: A pointer that is passed to the migrate function. If any + * objects cannot be touched at this point then the pointer may + * indicate a failure and then the migration function can simply + * remove the references that were already obtained. The private + * data could be used to track the objects that were already pinned. + */ +typedef void *kmem_cache_isolate_func(struct kmem_cache *s, void **ptr, int nr); + +/** + * typedef kmem_cache_migrate_func - Object migration callback function. + * @s: The cache we are working on. + * @ptr: Pointer to an array of pointers to the objects to migrate. + * @nr: Number of objects in @ptr array. + * @node: The NUMA node where the object should be allocated. + * @private: The pointer returned by kmem_cache_isolate_func(). + * + * This function is responsible for migrating objects. Typically, for + * each object in the input array you will want to allocate an new + * object, copy the original object, update any pointers, and free the + * old object. + * + * After this function returns all pointers to the old object should now + * point to the new object. + * + * Context: Called with no locks held and interrupts enabled. Sleeping + * is possible. Any operation may be performed. + */ +typedef void kmem_cache_migrate_func(struct kmem_cache *s, void **ptr, + int nr, int node, void *private); + +/* + * kmem_cache_setup_mobility() is used to setup callbacks for a slab cache. + */ +#ifdef CONFIG_SLUB +void kmem_cache_setup_mobility(struct kmem_cache *, kmem_cache_isolate_func, + kmem_cache_migrate_func); +#else +static inline void +kmem_cache_setup_mobility(struct kmem_cache *s, kmem_cache_isolate_func isolate, + kmem_cache_migrate_func migrate) {} +#endif + /* * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above. diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index d2153789bd9f..2879a2f5f8eb 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -99,6 +99,9 @@ struct kmem_cache { gfp_t allocflags; /* gfp flags to use on each alloc */ int refcount; /* Refcount for slab cache destroy */ void (*ctor)(void *); + kmem_cache_isolate_func *isolate; + kmem_cache_migrate_func *migrate; + unsigned int inuse; /* Offset to metadata */ unsigned int align; /* Alignment */ unsigned int red_left_pad; /* Left redzone padding size */ diff --git a/mm/slub.c b/mm/slub.c index cd04dbd2b5d0..1c380a2bc78a 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4317,6 +4317,33 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return err; } +void kmem_cache_setup_mobility(struct kmem_cache *s, + kmem_cache_isolate_func isolate, + kmem_cache_migrate_func migrate) +{ + /* + * Mobile objects must have a ctor otherwise the object may be + * in an undefined state on allocation. Since the object may + * need to be inspected by the migration function at any time + * after allocation we must ensure that the object always has a + * defined state. + */ + if (!s->ctor) { + pr_err("%s: require constructor to setup mobility\n", s->name); + return; + } + + s->isolate = isolate; + s->migrate = migrate; + + /* + * Sadly serialization requirements currently mean that we have + * to disable fast cmpxchg based processing. + */ + s->flags &= ~__CMPXCHG_DOUBLE; +} +EXPORT_SYMBOL(kmem_cache_setup_mobility); + void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller) { struct kmem_cache *s; @@ -5001,13 +5028,33 @@ static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf, } SLAB_ATTR(cpu_partial); -static ssize_t ctor_show(struct kmem_cache *s, char *buf) +static int op_show(char *buf, const char *txt, unsigned long addr) { - if (!s->ctor) - return 0; - return sprintf(buf, "%pS\n", s->ctor); + int x = 0; + + x += sprintf(buf, "%s : ", txt); + x += sprint_symbol(buf + x, addr); + x += sprintf(buf + x, "\n"); + + return x; +} + +static ssize_t ops_show(struct kmem_cache *s, char *buf) +{ + int x = 0; + + if (s->ctor) + x += op_show(buf + x, "ctor", (unsigned long)s->ctor); + + if (s->isolate) + x += op_show(buf + x, "isolate", (unsigned long)s->isolate); + + if (s->migrate) + x += op_show(buf + x, "migrate", (unsigned long)s->migrate); + + return x; } -SLAB_ATTR_RO(ctor); +SLAB_ATTR_RO(ops); static ssize_t aliases_show(struct kmem_cache *s, char *buf) { @@ -5420,7 +5467,7 @@ static struct attribute *slab_attrs[] = { &objects_partial_attr.attr, &partial_attr.attr, &cpu_slabs_attr.attr, - &ctor_attr.attr, + &ops_attr.attr, &aliases_attr.attr, &align_attr.attr, &hwcache_align_attr.attr, From patchwork Mon May 20 05:40:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949893 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 68FCF16C1 for ; Mon, 20 May 2019 05:41:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 581A82018F for ; Mon, 20 May 2019 05:41:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4B7932864B; Mon, 20 May 2019 05:41:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98B902022C for ; Mon, 20 May 2019 05:41:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C8DA6B0007; Mon, 20 May 2019 01:41:26 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 97A5F6B0008; Mon, 20 May 2019 01:41:26 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 840D36B000A; Mon, 20 May 2019 01:41:26 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 62C6D6B0007 for ; Mon, 20 May 2019 01:41:26 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id c48so13232543qta.19 for ; Sun, 19 May 2019 22:41:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=bxV5jBaXvA/LKL0qOR0ZLSK5IIfdrfWAd0nyjsopfiM=; b=jJKfB7FF/bEm528nfEze5SZ5Rf6jHoBgY4pGXsKHdWWZXlfAeh7hCeUTJgDt/z8cPi SEsUhQ6N9ICSqQeQQyrdd8RieH9XGWw4UucboyejCA+eM8bNyNbo5+PHFtFl/nyliOyz KRrLctSNMODDvLk2Zu/MGB8GHHNJRct/+IDdzPSEIs+uAXPexLh6h9An+bU8AOb2g2r+ 4nN3I7Ktppbe1XSlXEDAbobfAa0kZ8sdFByEhi2BNsR+PHLMmTx4MD8XIuSqNYV+4Yav vSx09CnGIyUXdoLHtCK7X7bDkxXlvVBAZbSE/0R6aB7fRb4eFwf+/5ccF5Y7u4AJTZBX WARQ== X-Gm-Message-State: APjAAAUVhUyGp884JoFsNq8s7tOhz8ieLxaQCTIrSabPjGjJ1oyeibN8 C2gW6zyzZ0SKywOJrCk9IiGCESF9YklQH8x7InpuGqgNtCGma5jNtvNEcZnx/SE9+zrGq6DE/Q6 4D0DWrMu53voFDmEEjfiNr0C3APFdYGg0HhRkupXjUjmqBdoGBG3c/uxJlRagAVA= X-Received: by 2002:ae9:ef0e:: with SMTP id d14mr56059512qkg.232.1558330886143; Sun, 19 May 2019 22:41:26 -0700 (PDT) X-Google-Smtp-Source: APXvYqwr6s6UdAMNbzHUSPSpy4y3RENakWgzOGqtMytCEzRBNRTxfrMVU23fV6UqEk4S3NyY08PB X-Received: by 2002:ae9:ef0e:: with SMTP id d14mr56059458qkg.232.1558330884968; Sun, 19 May 2019 22:41:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330884; cv=none; d=google.com; s=arc-20160816; b=QVnDMSUtsR+Veyl7R6vJ1q1TIczjWZpGVtIWTDKJAa9NkRCRYg1VNBVxbFRW6ymHC/ AJQ4D73gm5D0QyIecpYFDzTRcUcizvsW+G2rrq/jXjiNXOaE5TS4fA8f1Z30yomVYCJL jnAU2drGF8UYgwaDdjKM75IWV56ojmidcAX+dColGFyNLm29lNZKKwLn2cv8dr/sPZsf q8Jm5E1jwDWH061Gfo6yX+WJ2yeFCxOND4J8a7mZf46nRBfJnhwwwtaI8WHj34uGgULe gXbhN47FoMQzL/bPVSVZmWo+xzZMDiTFgb4GKIKuXS1ZjCurSTGso4BzbKaMzAjcHa8H 2kEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=bxV5jBaXvA/LKL0qOR0ZLSK5IIfdrfWAd0nyjsopfiM=; b=tAXr4XYKD55/1/pdOSWmb8lvDmQ99kBPnuA0ke69oCFG1u9VZJATk5mAwm/j+t87SI m7Jt8n4E8XsZuWp4c7wdrbfnXbuoWaCd8CHmX+Z8NHrWCR+j/UM1Nfp8odXbk8UkmPeO AfgmPTYueS6kaSXb7HFwjwixJ6IBOviIdYe8fbbNZZpiglLYmvB1LcXHUDnZ+U/3CGaC hCX7BaGA0GH0UY6dZzf/dhURpcf5/7Likjy1LvEit39XIn6VfYi9fEoLROkD1tQRyTTQ auvvLUa3iVaE1Y0Zmg0pMZZ3alP+eqJyZN4Va/emb/iQel2MtuTY+qY+wRBFA9vVSlLJ qRng== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=brPSfUHt; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id t39si2121458qvc.158.2019.05.19.22.41.24 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:41:24 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=brPSfUHt; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id AA07F3242; Mon, 20 May 2019 01:41:24 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:41:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=bxV5jBaXvA/LKL0qOR0ZLSK5IIfdrfWAd0nyjsopfiM=; b=brPSfUHt HQXU8KIV4VyCcC+GuC0fXcMB0eb698X1lziIGBJKJQV+YM0rADkU+JlSFxcWmKFi 2MvgcDOFDrFrtRghaKvg2pyJOqsgBJZkrVokqUkVPf5EcMEOvXsYusP+ohcAt44n k5ewq8PKHTVPqBh4ak1ePk9Ap48aMDgQ3M6OLTvltT685wf9sOeeVNGqt62rDrY3 JgSAMVJ7+ySQVET1J/Emzh4sONr3oIrcb8wwids6WJcwYnZBmFkgLX+ABVPfktH0 Vd1Q9emQYKND0WC/SM4z1wFr00TiHpnRP0lI1FZOXPr6SDDl92Io1SDXwnriXIe5 ftkCG63PUufXHQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepud X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 81E548005B; Mon, 20 May 2019 01:41:17 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 02/16] tools/vm/slabinfo: Add support for -C and -M options Date: Mon, 20 May 2019 15:40:03 +1000 Message-Id: <20190520054017.32299-3-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP -C lists caches that use a ctor. -M lists caches that support object migration. Add command line options to show caches with a constructor and caches that are movable (i.e. have migrate function). Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 40 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index 73818f1b2ef8..cbfc56c44c2f 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -33,6 +33,7 @@ struct slabinfo { unsigned int hwcache_align, object_size, objs_per_slab; unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; + int movable, ctor; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; unsigned long free_fastpath, free_slowpath; @@ -67,6 +68,8 @@ int show_report; int show_alias; int show_slab; int skip_zero = 1; +int show_movable; +int show_ctor; int show_numa; int show_track; int show_first_alias; @@ -109,11 +112,13 @@ static void fatal(const char *x, ...) static void usage(void) { - printf("slabinfo 4/15/2011. (c) 2007 sgi/(c) 2011 Linux Foundation.\n\n" - "slabinfo [-aADefhilnosrStTvz1LXBU] [N=K] [-dafzput] [slab-regexp]\n" + printf("slabinfo 4/15/2017. (c) 2007 sgi/(c) 2011 Linux Foundation/(c) 2017 Jump Trading LLC.\n\n" + "slabinfo [-aACDefhilMnosrStTvz1LXBU] [N=K] [-dafzput] [slab-regexp]\n" + "-a|--aliases Show aliases\n" "-A|--activity Most active slabs first\n" "-B|--Bytes Show size in bytes\n" + "-C|--ctor Show slabs with ctors\n" "-D|--display-active Switch line format to activity\n" "-e|--empty Show empty slabs\n" "-f|--first-alias Show first alias\n" @@ -121,6 +126,7 @@ static void usage(void) "-i|--inverted Inverted list\n" "-l|--slabs Show slabs\n" "-L|--Loss Sort by loss\n" + "-M|--movable Show caches that support movable objects\n" "-n|--numa Show NUMA information\n" "-N|--lines=K Show the first K slabs\n" "-o|--ops Show kmem_cache_ops\n" @@ -588,6 +594,12 @@ static void slabcache(struct slabinfo *s) if (show_empty && s->slabs) return; + if (show_ctor && !s->ctor) + return; + + if (show_movable && !s->movable) + return; + if (sort_loss == 0) store_size(size_str, slab_size(s)); else @@ -602,6 +614,10 @@ static void slabcache(struct slabinfo *s) *p++ = '*'; if (s->cache_dma) *p++ = 'd'; + if (s->ctor) + *p++ = 'C'; + if (s->movable) + *p++ = 'M'; if (s->hwcache_align) *p++ = 'A'; if (s->poison) @@ -636,7 +652,8 @@ static void slabcache(struct slabinfo *s) printf("%-21s %8ld %7d %15s %14s %4d %1d %3ld %3ld %s\n", s->name, s->objects, s->object_size, size_str, dist_str, s->objs_per_slab, s->order, - s->slabs ? (s->partial * 100) / s->slabs : 100, + s->slabs ? (s->partial * 100) / + (s->slabs * s->objs_per_slab) : 100, s->slabs ? (s->objects * s->object_size * 100) / (s->slabs * (page_size << s->order)) : 100, flags); @@ -1256,6 +1273,13 @@ static void read_slab_dir(void) slab->alloc_node_mismatch = get_obj("alloc_node_mismatch"); slab->deactivate_bypass = get_obj("deactivate_bypass"); chdir(".."); + if (read_slab_obj(slab, "ops")) { + if (strstr(buffer, "ctor :")) + slab->ctor = 1; + if (strstr(buffer, "migrate :")) + slab->movable = 1; + } + if (slab->name[0] == ':') alias_targets++; slab++; @@ -1332,6 +1356,8 @@ static void xtotals(void) } struct option opts[] = { + { "ctor", no_argument, NULL, 'C' }, + { "movable", no_argument, NULL, 'M' }, { "aliases", no_argument, NULL, 'a' }, { "activity", no_argument, NULL, 'A' }, { "debug", optional_argument, NULL, 'd' }, @@ -1367,7 +1393,7 @@ int main(int argc, char *argv[]) page_size = getpagesize(); - while ((c = getopt_long(argc, argv, "aAd::Defhil1noprstvzTSN:LXBU", + while ((c = getopt_long(argc, argv, "aACd::Defhil1MnoprstvzTSN:LXBU", opts, NULL)) != -1) switch (c) { case '1': @@ -1376,6 +1402,9 @@ int main(int argc, char *argv[]) case 'a': show_alias = 1; break; + case 'C': + show_ctor = 1; + break; case 'A': sort_active = 1; break; @@ -1399,6 +1428,9 @@ int main(int argc, char *argv[]) case 'i': show_inverted = 1; break; + case 'M': + show_movable = 1; + break; case 'n': show_numa = 1; break; From patchwork Mon May 20 05:40:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949897 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12D7616C1 for ; Mon, 20 May 2019 05:41:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 02337205FC for ; Mon, 20 May 2019 05:41:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA2CB28647; Mon, 20 May 2019 05:41:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6E2752864B for ; Mon, 20 May 2019 05:41:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 730716B0008; Mon, 20 May 2019 01:41:33 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6E2656B000A; Mon, 20 May 2019 01:41:33 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D00D6B000C; Mon, 20 May 2019 01:41:33 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 3BC016B0008 for ; Mon, 20 May 2019 01:41:33 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id h16so11746395qke.11 for ; Sun, 19 May 2019 22:41:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=pvmcntUKaLFWeIKV28cwZMSwwZ0f8GZCwK0RuRd6vxw=; b=qpx0NjPqJOUI0iQycBFNNrE8aCezSgwivPEqhpOgK3E2Jn2cIHGbq4WjsO/gwM5y03 kNNofMXJ+PKbJvks6rYkmCRz5rvUw6voIRyFUxKHM4iSkOklf1MgasFsbOh5OtiPG+RI hSaeEj+m1/2bJ4COb3XwhaoXoH7/Jkl3iKW6C7tpxEAHzarmkqE4vVm54iJZLQfV/sjd VZ+d9mdxbbR7usD9bfd8ntD6jVNUaw/cMtFMLeoexHmxHll0rFutBJl5YxaBsx9wJLtI SAEweasbKyQE8qilzI0Sxc7CsfoCACotB/3NUowg3IYnoheeV1aASqeiN9o1cRPe51O/ pkNg== X-Gm-Message-State: APjAAAUDTM+LRK/P6OmmvYQ+zfp2rbl/RWHzfW7rScS8Y9MpQUSrQhFb I2LiucAXwil3rVOX/iCzrP8TTBFug9b51+iLu1Jn7iWHxPrbwLwKOHLHqxS1K89hl+6YXQQEYL4 /gRLwI9JB5ZqVLAt38hazufTXhO1xm//02GCBejXqNdJqoiacIm+VyeRP/giE1+k= X-Received: by 2002:aed:2428:: with SMTP id r37mr61067749qtc.213.1558330893006; Sun, 19 May 2019 22:41:33 -0700 (PDT) X-Google-Smtp-Source: APXvYqxwdr5k3/QdFzYRbuHLDqP5z38MCtyO5IGYGPYZskZMCsRnBiwgUi/O0dOuFszTKiAE+YZw X-Received: by 2002:aed:2428:: with SMTP id r37mr61067712qtc.213.1558330892151; Sun, 19 May 2019 22:41:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330892; cv=none; d=google.com; s=arc-20160816; b=C1b2EZYopxKyJ/dinhTyzq4SbsXf5+DC9ysKBzD5a3mqJFaeqKfiixHeHAFZX6lZjn DvdhBnTOVRjraK/F8AunSVlfTZIIVGP2U5gYG5gScTov4YlTzcTymW4KtR2ePR/+xuJg K74wYvS3jWpf0237ml0VozBMV0BfySwmshJQ79M8WPcKGX57zB/lYV+QpslJY94jNWk4 uSpMOX62MLy7nyrX0iza/DarcjECJ/gfkZhz1E1y8SY4Nd+XoHKEkX9uS0XUgG3zGtsT N7NCjo3Ib6U/KIueuIx6EiXwIMLRvu5v+9vMq71GyRBY3/WfeN19Of3ZKvyhV/b7Jy69 lLKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=pvmcntUKaLFWeIKV28cwZMSwwZ0f8GZCwK0RuRd6vxw=; b=ygemhDaFYPHgr91NTbs0SVN9O9yaqnViFZU7V25vykq+GsY8663/8B46FqWL+uxKyZ 4HGgWpw8rNIPmAcvHchwpCo/VuGQawDSu/ev94MlVvBIAgUkD4uToHmPC58rHWdUV2Vr jzKmAxyJXRbPamdXjuQfTQN3G0JEtM9XPwt7UQATqDcF89WHlhTuckh9RQS8OIb0fxju 1qJX5P24kcq3XfdCy5uwvlxHe4pHdoP+b5wJbOg36NpImpZIv780sH4i5+ZUcP4La3M8 Dgt52ib9kkAUQF08wBhGVwuXScTrUmYpyALGfaMoLvKNCDp4H7W4cRfQaE8nQYmiC2MY qrGw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="Tp/kx4vO"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id p37si3008321qvc.125.2019.05.19.22.41.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:41:32 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="Tp/kx4vO"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id D6EB41160B; Mon, 20 May 2019 01:41:31 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:41:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=pvmcntUKaLFWeIKV28cwZMSwwZ0f8GZCwK0RuRd6vxw=; b=Tp/kx4vO UsH3lifxwz1hf6IIZvoCuYCbS3N/bxeIEVJczgn7f7NSK3ljq+hzZo75SxFHhs61 eLKytTBmVNQeJfPJx0AqskyqdiYk4kRZD/sDasZgyVCdPfTew11AAqBLFyvz3ZiL gEoATFHVziwmgiSWPSPgNIBzkGFdMSk/oKmmb9a9XNpVoRjO6Eg08rYHYIw93/h6 avf2LrUg+U9Sj6kHhjdlMRyWFY3YosBdV2e46UtDbn3yVm3YtKtj2mHDUyv0SF41 tWoioUGzJWBWyPx/TgE7scZXI6laUj3o5rFiu1lZOANRvRPYxV6XVkLIO1yKT8El xjvjA4jOCg5H/w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepvd X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id BA1B18005B; Mon, 20 May 2019 01:41:24 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 03/16] slub: Sort slab cache list Date: Mon, 20 May 2019 15:40:04 +1000 Message-Id: <20190520054017.32299-4-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It is advantageous to have all defragmentable slabs together at the beginning of the list of slabs so that there is no need to scan the complete list. Put defragmentable caches first when adding a slab cache and others last. Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding Reviewed-by: Roman Gushchin --- mm/slab_common.c | 2 +- mm/slub.c | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..db5e9a0b1535 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -393,7 +393,7 @@ static struct kmem_cache *create_cache(const char *name, goto out_free_cache; s->refcount = 1; - list_add(&s->list, &slab_caches); + list_add_tail(&s->list, &slab_caches); memcg_link_cache(s); out: if (err) diff --git a/mm/slub.c b/mm/slub.c index 1c380a2bc78a..66d474397c0f 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4333,6 +4333,8 @@ void kmem_cache_setup_mobility(struct kmem_cache *s, return; } + mutex_lock(&slab_mutex); + s->isolate = isolate; s->migrate = migrate; @@ -4341,6 +4343,10 @@ void kmem_cache_setup_mobility(struct kmem_cache *s, * to disable fast cmpxchg based processing. */ s->flags &= ~__CMPXCHG_DOUBLE; + + list_move(&s->list, &slab_caches); /* Move to top */ + + mutex_unlock(&slab_mutex); } EXPORT_SYMBOL(kmem_cache_setup_mobility); From patchwork Mon May 20 05:40:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949901 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7BD9A1390 for ; Mon, 20 May 2019 05:41:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A73A2863C for ; Mon, 20 May 2019 05:41:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5E6B528630; Mon, 20 May 2019 05:41:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2A5528644 for ; Mon, 20 May 2019 05:41:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C8B46B000A; Mon, 20 May 2019 01:41:41 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 979796B000C; Mon, 20 May 2019 01:41:41 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 867EE6B000D; Mon, 20 May 2019 01:41:41 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 5EB376B000A for ; Mon, 20 May 2019 01:41:41 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id n5so6698966qkf.7 for ; Sun, 19 May 2019 22:41:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=N2uiMeBCEcpu2BPOPPXFXHxvTcG1otfwurF2oyJcOqs=; b=Q2vlddu08qGTeu2B7FQ6+uUqJbDunSnhRD4QOrLjUGG5Lltrup9+o+03rNxmZrMU3x Twwvm6B6KUjtUVm9glIbRsGxmoWS+MNk/NmHZ+O3sgGFUjqMHKMUvi9oOGq+/mpsgYaW +AA5EYZ6gBNME4Em06EkKE/FyBsovfwi6Aoz5vQcj2sw/05Vp+TAvyHxxQ5bMyMk3k7q 8gMFKGZvpOjVNX/zQ0GJFgm7RErpGiML2SlzL4WsLWjm1EWRjaeV5XmuN6yhzirj8XZJ v226avF9f/E9XnGWTeuLkwFuRZ/k54shfpO63QaWEbDLMlj9JixdyuvozGlv7xjg3LaY 3Srw== X-Gm-Message-State: APjAAAWhIw7tJv6ZSZqtQ5TwDmNvTmt4E8WpkxV5Xp2d8zfbiU9/DBAz hZrSQH+zdx68Wy7kCz5GvP5jvyJSb0/3yzaBY9ITvhIbBOBPlHev75u6Vukhrhf1/GXs2gJB69I ujb9fWXUT30JHn4HoGeIr0CuIjCCeoTXjyPY87wC91L8nnQxlC5iy1ONWy9Ga3eI= X-Received: by 2002:a0c:87f4:: with SMTP id 49mr58501021qvk.149.1558330901040; Sun, 19 May 2019 22:41:41 -0700 (PDT) X-Google-Smtp-Source: APXvYqzLyRpFfR9joe0F8QFKX3MzncYQktsbVDfbf8AtmFwdpCId7weQkw/ufw2FifGRfjwovZOC X-Received: by 2002:a0c:87f4:: with SMTP id 49mr58500953qvk.149.1558330899652; Sun, 19 May 2019 22:41:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330899; cv=none; d=google.com; s=arc-20160816; b=aXmwRvYz7nUe6tZCILd8RMdkMiz3tjJgJJ0AjFsA0hAtqBLZyCbcO8Sti3nIhB7k7K dqgaqJOnGo5xJ9RQcicE2rLfMVIvsStVTnQnczvUPfQ8FwXuD59XBpUiEFzm5QXIOkID BHebhasYzh/TlBF6fi28V7HcwXJq9nFUnk2VkfORr4H7xFSkwBz65qorvbGjtNlkBeVk teDpSP5bicPsvZXGwTGbfq/5GT6cbBW25v1MQFzafr6jlUWwx5h/+uxGSvoMibwsSIz+ h+gpg/7jVw9e09CUhuWeZ0HarCd9IHdChUZqWWRiICZAnnTxTUKZoU9vTnOSHDUP8sci mNtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=N2uiMeBCEcpu2BPOPPXFXHxvTcG1otfwurF2oyJcOqs=; b=s8SGb1xLay7fvQOw1BImyiC4m0FoyrjtXFHwxJOTrzqlIjAEpaP/o3nHF3bX2Fes8F sCbTgFO5AXlhVKLxzVoMqoZXc9Bp5ihtOvAsUH0zjtss7Syp4DKSsI3pd3XHsBbyKLRJ 0YtZcdDkzGVkKeNRlhk0AQf9fnmqqHU90zmRGm51dD2WUgqcduwdNupYLIgOXpczvHtN qIVs0Q+/qOf+ec0R/SpZF3ixAZJGQogvO5MxEwOgLYCDRfi0jlH83nNvJwQFR8yhceub gN09VonxdU7iL2V6eNwF4VMCmNfVNd5gPCKuoIY2GVMl/FA0oHLcnQeQTphv6RVPfZNO +M4w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=NA4ek0DU; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id 28si1261498qtx.181.2019.05.19.22.41.39 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:41:39 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=NA4ek0DU; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 5DB4C289E; Mon, 20 May 2019 01:41:39 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:41:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=N2uiMeBCEcpu2BPOPPXFXHxvTcG1otfwurF2oyJcOqs=; b=NA4ek0DU mjDa+hddR537O0RJrTOoq4ljhflX0qakO6lclAhdYk2Br9meznoxfQdnhx++lqFq 88fvI36MkodEIPVH+kYGQ6I4ZaRjeF84TnE9sOTndxppbR1LsBOSvMUUc7GWr3Rl MU/dtDIl4h29kzM+966TNJvtrZIHxWs4DXGOgPYGKaGz+UTaWwxK9nGKwN4/xczf w1A50CBCUXApVv7CcmRuidgyl3Z6yexCEsOZyTZ8l8GeDkXZ59FT3evg3EiqzSSY 1RwYTxPwaIdVSpgnQdphsw4bVVkJlAJ5umpSDZ1eRXecJYN0rzo1yuRRirNj1IGe GJsDBkAzW6+5sQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepfe X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id EFCDC8005B; Mon, 20 May 2019 01:41:31 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 04/16] slub: Slab defrag core Date: Mon, 20 May 2019 15:40:05 +1000 Message-Id: <20190520054017.32299-5-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Internal fragmentation can occur within pages used by the slub allocator. Under some workloads large numbers of pages can be used by partial slab pages. This under-utilisation is bad simply because it wastes memory but also because if the system is under memory pressure higher order allocations may become difficult to satisfy. If we can defrag slab caches we can alleviate these problems. Implement Slab Movable Objects in order to defragment slab caches. Slab defragmentation may occur: 1. Unconditionally when __kmem_cache_shrink() is called on a slab cache by the kernel calling kmem_cache_shrink(). 2. Unconditionally through the use of the slabinfo command. slabinfo -s 3. Conditionally via the use of kmem_cache_defrag() - Use Slab Movable Objects when shrinking cache. Currently when the kernel calls kmem_cache_shrink() we curate the partial slabs list. If object migration is not enabled for the cache we still do this, if however, SMO is enabled we attempt to move objects in partially full slabs in order to defragment the cache. Shrink attempts to move all objects in order to reduce the cache to a single partial slab for each node. - Add conditional per node defrag via new function: kmem_defrag_slabs(int node). kmem_defrag_slabs() attempts to defragment all slab caches for node. Defragmentation is done conditionally dependent on MAX_PARTIAL _and_ defrag_used_ratio. Caches are only considered for defragmentation if the number of partial slabs exceeds MAX_PARTIAL (per node). Also, defragmentation only occurs if the usage ratio of the slab is lower than the configured percentage (sysfs field added in this patch). Fragmentation ratios are measured by calculating the percentage of objects in use compared to the total number of objects that the slab page can accommodate. The scanning of slab caches is optimized because the defragmentable slabs come first on the list. Thus we can terminate scans on the first slab encountered that does not support defragmentation. kmem_defrag_slabs() takes a node parameter. This can either be -1 if defragmentation should be performed on all nodes, or a node number. Defragmentation may be disabled by setting defrag ratio to 0 echo 0 > /sys/kernel/slab//defrag_used_ratio - Add a defrag ratio sysfs field and set it to 30% by default. A limit of 30% specifies that more than 3 out of 10 available slots for objects need to be in use otherwise slab defragmentation will be attempted on the remaining objects. In order for a cache to be defragmentable the cache must support object migration (SMO). Enabling SMO for a cache is done via a call to the recently added function: void kmem_cache_setup_mobility(struct kmem_cache *, kmem_cache_isolate_func, kmem_cache_migrate_func); Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- Documentation/ABI/testing/sysfs-kernel-slab | 14 + include/linux/slab.h | 1 + include/linux/slub_def.h | 7 + mm/slub.c | 385 ++++++++++++++++---- 4 files changed, 334 insertions(+), 73 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-kernel-slab b/Documentation/ABI/testing/sysfs-kernel-slab index 29601d93a1c2..c6f129af035a 100644 --- a/Documentation/ABI/testing/sysfs-kernel-slab +++ b/Documentation/ABI/testing/sysfs-kernel-slab @@ -180,6 +180,20 @@ Description: list. It can be written to clear the current count. Available when CONFIG_SLUB_STATS is enabled. +What: /sys/kernel/slab/cache/defrag_used_ratio +Date: May 2019 +KernelVersion: 5.2 +Contact: Christoph Lameter + Pekka Enberg , +Description: + The defrag_used_ratio file allows the control of how aggressive + slab fragmentation reduction works at reclaiming objects from + sparsely populated slabs. This is a percentage. If a slab has + less than this percentage of objects allocated then reclaim will + attempt to reclaim objects so that the whole slab page can be + freed. 0% specifies no reclaim attempt (defrag disabled), 100% + specifies attempt to reclaim all pages. The default is 30%. + What: /sys/kernel/slab/cache/deactivate_to_tail Date: February 2008 KernelVersion: 2.6.25 diff --git a/include/linux/slab.h b/include/linux/slab.h index 886fc130334d..4bf381b34829 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -149,6 +149,7 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, void (*ctor)(void *)); void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); +unsigned long kmem_defrag_slabs(int node); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index 2879a2f5f8eb..34c6f1250652 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -107,6 +107,13 @@ struct kmem_cache { unsigned int red_left_pad; /* Left redzone padding size */ const char *name; /* Name (only for display!) */ struct list_head list; /* List of slab caches */ + int defrag_used_ratio; /* + * Ratio used to check against the + * percentage of objects allocated in a + * slab page. If less than this ratio + * is allocated then reclaim attempts + * are made. + */ #ifdef CONFIG_SYSFS struct kobject kobj; /* For sysfs */ struct work_struct kobj_remove_work; diff --git a/mm/slub.c b/mm/slub.c index 66d474397c0f..2157205df7ba 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -355,6 +355,12 @@ static __always_inline void slab_lock(struct page *page) bit_spin_lock(PG_locked, &page->flags); } +static __always_inline int slab_trylock(struct page *page) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + return bit_spin_trylock(PG_locked, &page->flags); +} + static __always_inline void slab_unlock(struct page *page) { VM_BUG_ON_PAGE(PageTail(page), page); @@ -3634,6 +3640,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) set_cpu_partial(s); + s->defrag_used_ratio = 30; #ifdef CONFIG_NUMA s->remote_node_defrag_ratio = 1000; #endif @@ -3950,79 +3957,6 @@ void kfree(const void *x) } EXPORT_SYMBOL(kfree); -#define SHRINK_PROMOTE_MAX 32 - -/* - * kmem_cache_shrink discards empty slabs and promotes the slabs filled - * up most to the head of the partial lists. New allocations will then - * fill those up and thus they can be removed from the partial lists. - * - * The slabs with the least items are placed last. This results in them - * being allocated from last increasing the chance that the last objects - * are freed in them. - */ -int __kmem_cache_shrink(struct kmem_cache *s) -{ - int node; - int i; - struct kmem_cache_node *n; - struct page *page; - struct page *t; - struct list_head discard; - struct list_head promote[SHRINK_PROMOTE_MAX]; - unsigned long flags; - int ret = 0; - - flush_all(s); - for_each_kmem_cache_node(s, node, n) { - INIT_LIST_HEAD(&discard); - for (i = 0; i < SHRINK_PROMOTE_MAX; i++) - INIT_LIST_HEAD(promote + i); - - spin_lock_irqsave(&n->list_lock, flags); - - /* - * Build lists of slabs to discard or promote. - * - * Note that concurrent frees may occur while we hold the - * list_lock. page->inuse here is the upper limit. - */ - list_for_each_entry_safe(page, t, &n->partial, slab_list) { - int free = page->objects - page->inuse; - - /* Do not reread page->inuse */ - barrier(); - - /* We do not keep full slabs on the list */ - BUG_ON(free <= 0); - - if (free == page->objects) { - list_move(&page->slab_list, &discard); - n->nr_partial--; - } else if (free <= SHRINK_PROMOTE_MAX) - list_move(&page->slab_list, promote + free - 1); - } - - /* - * Promote the slabs filled up most to the head of the - * partial list. - */ - for (i = SHRINK_PROMOTE_MAX - 1; i >= 0; i--) - list_splice(promote + i, &n->partial); - - spin_unlock_irqrestore(&n->list_lock, flags); - - /* Release empty slabs */ - list_for_each_entry_safe(page, t, &discard, slab_list) - discard_slab(s, page); - - if (slabs_node(s, node)) - ret = 1; - } - - return ret; -} - #ifdef CONFIG_MEMCG static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) { @@ -4317,6 +4251,287 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return err; } +/* + * Allocate a slab scratch space that is sufficient to keep pointers to + * individual objects for all objects in cache and also a bitmap for the + * objects (used to mark which objects are active). + */ +static inline void *alloc_scratch(struct kmem_cache *s) +{ + unsigned int size = oo_objects(s->max); + + return kmalloc(size * sizeof(void *) + + BITS_TO_LONGS(size) * sizeof(unsigned long), + GFP_KERNEL); +} + +/* + * move_slab_page() - Move all objects in the given slab. + * @page: The slab we are working on. + * @scratch: Pointer to scratch space. + * @node: The target node to move objects to. + * + * If the target node is not the current node then the object is moved + * to the target node. If the target node is the current node then this + * is an effective way of defragmentation since the current slab page + * with its object is exempt from allocation. + */ +static void move_slab_page(struct page *page, void *scratch, int node) +{ + unsigned long objects; + struct kmem_cache *s; + unsigned long flags; + unsigned long *map; + void *private; + int count; + void *p; + void **vector = scratch; + void *addr = page_address(page); + + local_irq_save(flags); + slab_lock(page); + + BUG_ON(!PageSlab(page)); /* Must be a slab page */ + BUG_ON(!page->frozen); /* Slab must have been frozen earlier */ + + s = page->slab_cache; + objects = page->objects; + map = scratch + objects * sizeof(void **); + + /* Determine used objects */ + bitmap_fill(map, objects); + for (p = page->freelist; p; p = get_freepointer(s, p)) + __clear_bit(slab_index(p, s, addr), map); + + /* Build vector of pointers to objects */ + count = 0; + memset(vector, 0, objects * sizeof(void **)); + for_each_object(p, s, addr, objects) + if (test_bit(slab_index(p, s, addr), map)) + vector[count++] = p; + + if (s->isolate) + private = s->isolate(s, vector, count); + else + /* Objects do not need to be isolated */ + private = NULL; + + /* + * Pinned the objects. Now we can drop the slab lock. The slab + * is frozen so it cannot vanish from under us nor will + * allocations be performed on the slab. However, unlocking the + * slab will allow concurrent slab_frees to proceed. So the + * subsystem must have a way to tell from the content of the + * object that it was freed. + * + * If neither RCU nor ctor is being used then the object may be + * modified by the allocator after being freed which may disrupt + * the ability of the migrate function to tell if the object is + * free or not. + */ + slab_unlock(page); + local_irq_restore(flags); + + /* Perform callback to move the objects */ + s->migrate(s, vector, count, node, private); +} + +/* + * kmem_cache_defrag() - Defragment node. + * @s: cache we are working on. + * @node: The node to move objects from. + * @target_node: The node to move objects to. + * @ratio: The defrag ratio (percentage, between 0 and 100). + * + * Release slabs with zero objects and try to call the migration function + * for slabs with less than the 'ratio' percentage of objects allocated. + * + * Moved objects are allocated on @target_node. + * + * Return: The number of partial slabs left on @node after the + * operation. + */ +static unsigned long kmem_cache_defrag(struct kmem_cache *s, + int node, int target_node, int ratio) +{ + struct kmem_cache_node *n = get_node(s, node); + struct page *page, *page2; + LIST_HEAD(move_list); + unsigned long flags; + + if (node == target_node && n->nr_partial <= 1) { + /* + * Trying to reduce fragmentation on a node but there is + * only a single or no partial slab page. This is already + * the optimal object density that we can reach. + */ + return n->nr_partial; + } + + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &n->partial, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + if (page->inuse) { + if (page->inuse > ratio * page->objects / 100) { + slab_unlock(page); + /* + * Skip slab because the object density + * in the slab page is high enough. + */ + continue; + } + + list_move(&page->lru, &move_list); + if (s->migrate) { + /* Stop page being considered for allocations */ + n->nr_partial--; + page->frozen = 1; + } + slab_unlock(page); + } else { /* Empty slab page */ + list_del(&page->lru); + n->nr_partial--; + slab_unlock(page); + discard_slab(s, page); + } + } + + if (!s->migrate) { + /* + * No defrag method. By simply putting the zaplist at + * the end of the partial list we can let them simmer + * longer and thus increase the chance of all objects + * being reclaimed. + */ + list_splice(&move_list, n->partial.prev); + } + + spin_unlock_irqrestore(&n->list_lock, flags); + + if (s->migrate && !list_empty(&move_list)) { + void **scratch = alloc_scratch(s); + if (scratch) { + /* Try to remove / move the objects left */ + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + } + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + /* + * Objects left in slab page, move it to the + * tail of the partial list to increase the + * chance that the freeing of the remaining + * objects will free the slab page. + */ + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); + } + + return n->nr_partial; +} + +/** + * kmem_defrag_slabs() - Defrag slab caches. + * @node: The node to defrag or -1 for all nodes. + * + * Defrag slabs conditional on the amount of fragmentation in a page. + * + * Return: The total number of partial slabs in migratable caches left + * on @node after the operation. + */ +unsigned long kmem_defrag_slabs(int node) +{ + struct kmem_cache *s; + unsigned long left = 0; + int nid; + + if (node >= MAX_NUMNODES) + return -EINVAL; + + /* + * kmem_defrag_slabs() may be called from the reclaim path which + * may be called for any page allocator alloc. So there is the + * danger that we get called in a situation where slub already + * acquired the slub_lock for other purposes. + */ + if (!mutex_trylock(&slab_mutex)) + return 0; + + list_for_each_entry(s, &slab_caches, list) { + /* + * Defragmentable caches come first. If the slab cache is + * not defragmentable then we can stop traversing the list. + */ + if (!s->migrate) + break; + + if (node >= 0) { + if (s->node[node]->nr_partial > MAX_PARTIAL) { + left += kmem_cache_defrag(s, node, node, + s->defrag_used_ratio); + } + continue; + } + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (s->node[nid]->nr_partial > MAX_PARTIAL) { + left += kmem_cache_defrag(s, nid, nid, + s->defrag_used_ratio); + } + } + } + mutex_unlock(&slab_mutex); + return left; +} +EXPORT_SYMBOL(kmem_defrag_slabs); + +/** + * __kmem_cache_shrink() - Shrink a cache. + * @s: The cache to shrink. + * + * Reduces the memory footprint of a slab cache by as much as possible. + * + * This works by: + * 1. Removing empty slabs from the partial list. + * 2. Migrating slab objects to denser slab pages if the slab cache + * supports migration. If not, reorganizing the partial list so that + * more densely allocated slab pages come first. + * + * Not called directly, called by kmem_cache_shrink(). + */ +int __kmem_cache_shrink(struct kmem_cache *s) +{ + int node; + int left = 0; + + flush_all(s); + for_each_node_state(node, N_NORMAL_MEMORY) + left += kmem_cache_defrag(s, node, node, 100); + + return left; +} +EXPORT_SYMBOL(__kmem_cache_shrink); + void kmem_cache_setup_mobility(struct kmem_cache *s, kmem_cache_isolate_func isolate, kmem_cache_migrate_func migrate) @@ -5168,6 +5383,29 @@ static ssize_t destroy_by_rcu_show(struct kmem_cache *s, char *buf) } SLAB_ATTR_RO(destroy_by_rcu); +static ssize_t defrag_used_ratio_show(struct kmem_cache *s, char *buf) +{ + return sprintf(buf, "%d\n", s->defrag_used_ratio); +} + +static ssize_t defrag_used_ratio_store(struct kmem_cache *s, + const char *buf, size_t length) +{ + unsigned long ratio; + int err; + + err = kstrtoul(buf, 10, &ratio); + if (err) + return err; + + if (ratio > 100) + return -EINVAL; + + s->defrag_used_ratio = ratio; + return length; +} +SLAB_ATTR(defrag_used_ratio); + #ifdef CONFIG_SLUB_DEBUG static ssize_t slabs_show(struct kmem_cache *s, char *buf) { @@ -5492,6 +5730,7 @@ static struct attribute *slab_attrs[] = { &validate_attr.attr, &alloc_calls_attr.attr, &free_calls_attr.attr, + &defrag_used_ratio_attr.attr, #endif #ifdef CONFIG_ZONE_DMA &cache_dma_attr.attr, From patchwork Mon May 20 05:40:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949905 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D024A17D2 for ; Mon, 20 May 2019 05:41:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BEC9F205FC for ; Mon, 20 May 2019 05:41:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B18852863C; Mon, 20 May 2019 05:41:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 339442863E for ; Mon, 20 May 2019 05:41:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1644F6B000C; Mon, 20 May 2019 01:41:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 116116B000D; Mon, 20 May 2019 01:41:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 003416B000E; Mon, 20 May 2019 01:41:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id D47E86B000C for ; Mon, 20 May 2019 01:41:47 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id q72so2464479qke.19 for ; Sun, 19 May 2019 22:41:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=SYKDcIUq9lsWtxb4kl133qqF3/tTOqLHlnWEhsDaptArWg43FyFUiwVLbPQyIXA7pq VILi5IyAwBmepjsEp/on8Du+RQxUxiZAe7eyJaOk4C2lJjW7zQIq9faqnRoD3Epy5a+3 DMiez7g2iNDEmTlkArPy5vdvdz6QZ7qNfSyKi0uCPE9V0ErQreL3ZnONSuyKdyG6Tc7J AKpi9OfOlB/wJfvnrMBjEU4gjTaetpgOT42WVYfihVwbglqkAVtv9B6A4y1ZevNYv/0x 8+lZuHgvSIO1KQ21efOJ8tNd/OIfyFAeAmLOK9vCPqzJPQsGxPIp8Pu5ieJCLUpco/rc /LGA== X-Gm-Message-State: APjAAAVjFc3U7Nwp/OzptRUY1HdRYcXaLS4QcJaDK11VLvbg4SNK9TsZ dTqSX/MFgU2L8MzfwlfXUf2pdeMHsymrBLAkWndVgpXQ9GCm08z6yAf2p714KbmyCvWodzZdawT +5vq/vpA1MbUNYBpqka+y9w90D6bCXWsKJHFaa3f5XefDFP3GqT0SCHT8ppmP3QE= X-Received: by 2002:a0c:d0d4:: with SMTP id b20mr27138609qvh.38.1558330907649; Sun, 19 May 2019 22:41:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqwiJ4uruPJ8GnVlhHIL7BRPidr8NKnDUhVy6jlKgOEE5Ls8pO+5Nr9/Bki2oUGnaQGX1Iab X-Received: by 2002:a0c:d0d4:: with SMTP id b20mr27138573qvh.38.1558330906799; Sun, 19 May 2019 22:41:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330906; cv=none; d=google.com; s=arc-20160816; b=X/V4l5gr/9KwnbDYKU8GibelOGBQcUSDieAH5El9h/okFaIF5AfWB34W2BFX6o2QPX gl5qJRLyKP14my0l5KDd8pAWPXzeYhUcBJhCVF0Vft9ztx30QtHgGaKFbxmjDlAel3VR uA+sqJ5zPG0Y9SSVsUqiQF557LaOWqy4jovFvrxFOBZfg1wC3JuQ0H3DG2BGWgX0+2Qc vGN4Bmv4l1+oQLRbu2mpHAeH/KM9AfjxwkdGOctYVdrXj4a9aoxoAa2keQCuz9BZROKD jDjUZoLDB8X8Hgl3zSEMM1EyJTGv6833VCAftzGhCfn3qYgkN6ziVopPd7UVy49t3rqu kSDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=sFRIzjzACrQ1m6FVMpngVbGAsPZp2XmmIA2B7QHBx4MGqJdVg8J9004fdA5TM9LavZ DXpoedqEIB+09Nxs/Cf41+gaD5pqEiOYyKH5nOS7Vf97sioN7Dy7yCJoKzp7Svle1PdT C/FuHXrM0xvuq4p/y+H2V+7XKz8n3YbtbMXEZrAgHM4MZn9UJ6Mey8GG+fMbzKew6V9r 8gjAAkist1jZi+QStJygMcqqN9+MW4opJDKhd0KECnOyDWOn/v04sNVpHTeYhljfIFJq Ak4RZjolIW0kVyC67IkJ9V7XkFJj8/91NfmBSZIlRUcBm2cV9TKpAA2GZ0JiqC6laFPK 9kng== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="rV/ImNkG"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id s51si1451965qtk.243.2019.05.19.22.41.46 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:41:46 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="rV/ImNkG"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 84B5C4220; Mon, 20 May 2019 01:41:46 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:41:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=rV/ImNkG ycR32d1zRF48xl4Dc39cUienWu0iKlmdVUToUwm52yebhEK1C11Ucvr7tt0z8qJm iAtm3aaFFTJywcEtPiGVKcMGQTjMGvJt/vJ2Ne4ejracT8MrL3WjCjadCF0r6EJN OVbo6k05gTSJkCgDCXtHKFKBRXrc8TZUX1ey8kVL4oeZx8GAHUC4hBe5kt6Ry44T wu3V/xWcFF2jMiS2AJ/MOwHmI81KsxXnHOyh12Ce+QJ6SCDx9R+7NM1MTD6e88WT uI7d2gzOXVe535PQKo5vvdm4fu/7prpiTDuCaEEKWodgyNsRUnQKHsbggHfxA0do zI51yaCTbT1iPg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepge X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 6F46A8005B; Mon, 20 May 2019 01:41:39 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 05/16] tools/vm/slabinfo: Add remote node defrag ratio output Date: Mon, 20 May 2019 15:40:06 +1000 Message-Id: <20190520054017.32299-6-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add output line for NUMA remote node defrag ratio. Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index cbfc56c44c2f..d2c22f9ee2d8 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -34,6 +34,7 @@ struct slabinfo { unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; int movable, ctor; + int remote_node_defrag_ratio; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; unsigned long free_fastpath, free_slowpath; @@ -377,6 +378,10 @@ static void slab_numa(struct slabinfo *s, int mode) if (skip_zero && !s->slabs) return; + if (mode) { + printf("\nNUMA remote node defrag ratio: %3d\n", + s->remote_node_defrag_ratio); + } if (!line) { printf("\n%-21s:", mode ? "NUMA nodes" : "Slab"); for(node = 0; node <= highest_node; node++) @@ -1272,6 +1277,8 @@ static void read_slab_dir(void) slab->cpu_partial_free = get_obj("cpu_partial_free"); slab->alloc_node_mismatch = get_obj("alloc_node_mismatch"); slab->deactivate_bypass = get_obj("deactivate_bypass"); + slab->remote_node_defrag_ratio = + get_obj("remote_node_defrag_ratio"); chdir(".."); if (read_slab_obj(slab, "ops")) { if (strstr(buffer, "ctor :")) From patchwork Mon May 20 05:40:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3A8291390 for ; Mon, 20 May 2019 05:41:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 288CF28630 for ; Mon, 20 May 2019 05:41:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1C54128649; Mon, 20 May 2019 05:41:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A29D928647 for ; Mon, 20 May 2019 05:41:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 730BB6B000D; Mon, 20 May 2019 01:41:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6E08E6B000E; Mon, 20 May 2019 01:41:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D04A6B0010; Mon, 20 May 2019 01:41:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 3DFDC6B000D for ; Mon, 20 May 2019 01:41:55 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id c48so13233296qta.19 for ; Sun, 19 May 2019 22:41:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=eXHm4W/drsal867H3zIYC52YFAlL4avr+ytjs+XJiivHmUOPnfA1x09vWSqO87gbXl folk0bsK7GrppNu+GqD4bd55QVBDpFFaJ7XRa15M7RIBnJmYcwrz+ZQlc7Qe3VeUngu/ uqOFyhSzjf8cF+UO5/oCdA8Mb7+RFdUZtaMw3mmdeMY3TDwVHkmxFXrgAVRFO/qtKVy2 d8fvV0gNInnbtL4OuMd+koYrH9/MI1R1BXX/ChGcp0vawSnwQDGihXu6P860HO2uF8v6 CATFt4wFCC1ye8nj6CrIxvicvkXxWOO8kGEOOmlVCzhtRtTKmmr3sNhZQ1kqmivVlLFQ 92Pg== X-Gm-Message-State: APjAAAVzmfvgBvmGYINammrje7YHlpZKGrsVvsZuKLY8NMoci5i7rrtl sKtih71Pqe1YyvCgHL27tnS2WnwxWF8OvCpc17bMhSZ8HUkniwHytipo5gB4ZlCsRSgs1b3LrNY nZWZwuAzypICizSSfTbVhe4F4Kgec+ilaJDLJCK1W7zP71bQaYAeXWH6/ifMEtmQ= X-Received: by 2002:ac8:198f:: with SMTP id u15mr62500907qtj.153.1558330914955; Sun, 19 May 2019 22:41:54 -0700 (PDT) X-Google-Smtp-Source: APXvYqyAdzbmNppiuSE250qfH5caUJN41q9Pjg72Aa+UjT7uRokzzYPebEFgH32xInOzaeK9vjJH X-Received: by 2002:ac8:198f:: with SMTP id u15mr62500866qtj.153.1558330914112; Sun, 19 May 2019 22:41:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330914; cv=none; d=google.com; s=arc-20160816; b=mrR0CYUCZ63GlSvg8NxP24gFvlSKU5fATYsG7R31G0uuBl3rTCBYxi7oHlENfP8uGw Ii8ingkaMFGfIV/QEIjpNJrT2sKwcQbdc+CwTJhAANnB1r5ZlxYK0eHT6NSn9t8AWfBT 099vBQQnhBg/3jl6xAooRfUbpMtTGdCSFoYw4NMZ2HPvAWvO1h31ZTSC4DGzexFqaMVz pQB7wb/WNMg/gB0/AMp4UC4WDz+IxQiNEJ1db328/jtPQRdLmaxa1v+lZE4HkGpNLG9e qxLrPy1YU2h01tQq6xbGVA2/s0d3Mdy89rVZlTZ0kdCsb96MT6mf70f6e/G4cio26bNv 7uNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=fFytD07fj8O2jrTJlG2tnlu3G8XFttJPbIQ/bPJE+wxhz6Hp/plX5xx+GUt6x3Wzp/ ZKmWUoIU4zLNz0mDSAJRpmgqdVFv8bcx+Hft5oLehzE+SabZ6Oum1qUErVUqmTplPJj+ B2Jtj6tjm7Be1BzR3pHQcxBquWGYlvAhc1WWCSGjPoKSh7CFljb7IkUGJ7FEM1Besk+A ZlqXY16817u40pjzvdx8oqu5gLqjSZlM/U1SzSw2oGDc4LoG7GDbmbaWKo9hRsmjXTUx klTguE5BqWHeq1x6o3CWj56KDzFSKbLp5A7TlP4ufajJGHNrzaV4I9EK41VM0GOufmMh k1jw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=GbbDnSgm; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id e12si592470qve.129.2019.05.19.22.41.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:41:54 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=GbbDnSgm; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id CCF6846EF; Mon, 20 May 2019 01:41:53 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:41:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=GbbDnSgm 0kVZe8RNuEifClhviafSdIf6NVtk0GruV6Fn/MjenuqiXRGGoet1W8RyBsiTx2eJ KR2P/iazreJakJr9AVkSEzJA0WPSgdvAmKi0o6iNQjaVP0OlKwwfmIovSTbN0Yro HzUflNj+0yAYXx+hM1zie5xJ4bIc/Sn2+b7Hdl3qbH21xyoLdoPTB5JnAUyINPjw l7Iw5+ueEUGVCwHQ5nIJarApG0QFmrgTDpzjo+8+cypGNr6l8XmroCDGtVYwVDGQ gQLh5ZclTVb8T/VRLhmSSPQRWUKHrambSidQjQoY5/7RG7+bOd3EQSUQdSuzeOXc lJN1JwWphP9kJw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgephe X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id AEB008005B; Mon, 20 May 2019 01:41:46 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 06/16] tools/vm/slabinfo: Add defrag_used_ratio output Date: Mon, 20 May 2019 15:40:07 +1000 Message-Id: <20190520054017.32299-7-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add output for the newly added defrag_used_ratio sysfs knob. Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index d2c22f9ee2d8..ef4ff93df4cc 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -34,6 +34,7 @@ struct slabinfo { unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; int movable, ctor; + int defrag_used_ratio; int remote_node_defrag_ratio; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; @@ -549,6 +550,8 @@ static void report(struct slabinfo *s) printf("** Slabs are destroyed via RCU\n"); if (s->reclaim_account) printf("** Reclaim accounting active\n"); + if (s->movable) + printf("** Defragmentation at %d%%\n", s->defrag_used_ratio); printf("\nSizes (bytes) Slabs Debug Memory\n"); printf("------------------------------------------------------------------------\n"); @@ -1279,6 +1282,7 @@ static void read_slab_dir(void) slab->deactivate_bypass = get_obj("deactivate_bypass"); slab->remote_node_defrag_ratio = get_obj("remote_node_defrag_ratio"); + slab->defrag_used_ratio = get_obj("defrag_used_ratio"); chdir(".."); if (read_slab_obj(slab, "ops")) { if (strstr(buffer, "ctor :")) From patchwork Mon May 20 05:40:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 676D816C1 for ; Mon, 20 May 2019 05:42:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5353128635 for ; Mon, 20 May 2019 05:42:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4529F28647; Mon, 20 May 2019 05:42:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B9B0528635 for ; Mon, 20 May 2019 05:42:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6BAA96B000E; Mon, 20 May 2019 01:42:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 66CB56B0010; Mon, 20 May 2019 01:42:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50D386B0266; Mon, 20 May 2019 01:42:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 2B56E6B000E for ; Mon, 20 May 2019 01:42:04 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id c4so2150154qkd.16 for ; Sun, 19 May 2019 22:42:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=cDcQhkSl/wvS4nUzHJ8hWOnz6fnY6RBVhH8gnzcP0wMpnbz1c0eWZ7uqttyLtx5NHY pCh1F/ujUYVhA+YptzDGu0wYD0TbORQfwHgEqJGevP7VTiZ/KMd+tGlYVPs+bSt4KZuA 9r2z+ldGs+eXxytOD+60UOayy+DYCci8UHHkJsW/9qsMgrSTVQkhWJgSKeY+H/1w3bgP hiM4+isWYUtmylZnPncI30r6GyCiTMCMzjWGfj+bVkFdTqWvpKeeYgFCywQ3A0tAI3wH SllHk93ivgsYZ749hUnNHG21VewGeRo0m2D+92hqqTivKt4FQsQyzn9R0qH6R+tBCdkV 6BlA== X-Gm-Message-State: APjAAAXvFcLJi7aIBAWEHBCbA15wNBLZtlNTbPxb3ximSwtbLnk3T7Rv LycAnV9FhUs5VzUbJ/720h7f18p8EH+tvPluqhHvdr8GeHJU4A4cF+T3MZxW2SCk4is6mGR6x0S h3pqUuHdRdHJabkm0Pv0wM1aANXB58WuUxux9ZlpgOr53kosJ5zjf94SPpyHsFv4= X-Received: by 2002:ac8:17b1:: with SMTP id o46mr60313475qtj.71.1558330923793; Sun, 19 May 2019 22:42:03 -0700 (PDT) X-Google-Smtp-Source: APXvYqyHHnTJfmayuL7xMv+rqyJm3aOE29iGYPfffyFIS7ebeMHbTn5ymXKfrn80YLXpF72krovz X-Received: by 2002:ac8:17b1:: with SMTP id o46mr60313371qtj.71.1558330921651; Sun, 19 May 2019 22:42:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330921; cv=none; d=google.com; s=arc-20160816; b=wO69HEqT6Pn91jjGkozW+MB3m9wC9JfqamMT5EVtc+rbJSet8IvFU6IeYQWnpjdK3k iAw2tt4b/9ZsUMybR5AA/sYp2DKAqZ4SonU3ABKiF5P+hHkqlzFj9fBmOPM0jQVZil1s C2wSJ1r6VtZi6ZEu/bLTRYS+xmx2WAnwghNVIm4sx0Bq05d478+GQz1v9vXAtltrEKZs 1do3r8VzhllJEWS2xzijn2H0qis2iYbShEjcupvuhsGfqmy4NMHqE2Q4wH4Y9ckqeSF3 Y9/TMn7yYB+N9+0DpCiUj1CRBF29SL3r/1xETHLsFW+ctZxK1fUnQyRy/vICpdH29qXH wqFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=us9442lDpQIq8Kag56tjVO6rvlKkiDhns0nWVCrcLYntTbo+iITyAwxbiBaA6R8d9W iIQO/pwWSV4ko9vELOqJM5GMRE/NDvnaxx6jOU90hYpXBzrghcDOSvK3jd4N680fXt2V sRKLWPMOZePTIcWkB3IrS6IAY9DlkB+WECK2WDLufBOZ20T5zAVLQ8YUOivrIOK4m2J7 +BEEujRMIeV8uf43RlqOpg0oKghgHIYj4H5aB/cADs9XtBg99EFI9KzshNMHxFHw4gQa Es6s/Tb8aluZjTe5oBJJvkXJRnaQ7uEF3mwC/kR1hgbrxx+EIAoA0f6PxhZs3wrDK7te 4WCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=bs75uuyX; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id p1si2906685qkm.122.2019.05.19.22.42.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:42:01 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=bs75uuyX; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 5E20B9008; Mon, 20 May 2019 01:42:01 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:42:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=bs75uuyX fSFxxKyezZObsYSNl0tOiV5u+KhdQ7zhddTOu3JNCaeQwUFCKgHRH1QyJ3PoPUO4 /sHVTbEkL0ovNVRCRd/wWyLszvgzQ5n4WVnTgw1ZKutcmNwc3JJSsYObP96KOjvq cW7IYI3oQCDR+ccnd4vrTByTEB2cOes0qrLOFvSElYYcycY4/78WeQmoDFKC8uSn XfCup1KTAdOsRSbjEcifTfglZPWhlacvkc80s8IrjPXh5NOT0rfoxKE2qudlrqyc Gv8WkxqdLvPvUPJKuA3lH0q0RVFSDWBN6V/Ee6V0Cy/7oMh9cS6g9ZrR9ESM3IQG JeHMCj3TDCf4Zg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepie X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id E77618005B; Mon, 20 May 2019 01:41:53 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 07/16] tools/testing/slab: Add object migration test module Date: Mon, 20 May 2019 15:40:08 +1000 Message-Id: <20190520054017.32299-8-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just implemented slab movable objects for the SLUB allocator. We should test that code. In order to do so we need to be able to do a number of things - Create a cache - Enable Slab Movable Objects for the cache - Allocate objects to the cache - Free objects from within specific slabs of the cache We can do all this via a loadable module. Add a module that defines functions that can be triggered from userspace via a debugfs entry. From the source: /* * SLUB defragmentation a.k.a. Slab Movable Objects (SMO). * * This module is used for testing the SLUB allocator. Enables * userspace to run kernel functions via a debugfs file. * * debugfs: /sys/kernel/debugfs/smo/callfn (write only) * * String written to `callfn` is parsed by the module and associated * function is called. See fn_tab for mapping of strings to functions. */ References to allocated objects are kept by the module in a linked list so that userspace can control which object to free. We introduce the following four functions via the function table "enable": Enables object migration for the test cache. "alloc X": Allocates X objects "free X [Y]": Frees X objects starting at list position Y (default Y==0) "test": Runs [stress] tests from within the module (see below). {"enable", smo_enable_cache_mobility}, {"alloc", smo_alloc_objects}, {"free", smo_free_object}, {"test", smo_run_module_tests}, Freeing from the start of the list creates a hole in the slab being freed from (i.e. creates a partial slab). The results of running these commands can be see using `slabinfo` (available in tools/vm/): make -o slabinfo tools/vm/slabinfo.c Stress tests can be run from within the module. These tests are internal to the module because we verify that object references are still good after object migration. These are called 'stress' tests because it is intended that they create/free a lot of objects. Userspace can control the number of objects to create, default is 1000. Example test session -------------------- Relevant /proc/slabinfo column headers: name # mount -t debugfs none /sys/kernel/debug/ $ cd path/to/linux/tools/testing/slab; make ... # insmod slub_defrag.ko # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 0 0 392 20 2 From this we can see that the module created cache 'smo_test' with 20 objects per slab and 2 pages per slab (and cache is currently empty). We can play with the slab allocator manually: # insmod slub_defrag.ko # echo 'alloc 21' > callfn # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 21 40 392 20 2 We see here that 21 active objects have been allocated creating 2 slabs (40 total objects). # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 21 Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 2 Sanity Checks : On Total: 16384 SlabObj: 392 Full : 1 Redzoning : On Used : 1176 SlabSiz: 8192 Partial: 1 Poisoning : On Loss : 15208 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 7056 Align : 8 Objects: 20 Tracing : Off Lpadd: 704 Now free an object from the first slot of the first slab # echo 'free 1' > callfn # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 20 40 392 20 2 # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 20 Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 2 Sanity Checks : On Total: 16384 SlabObj: 392 Full : 0 Redzoning : On Used : 1120 SlabSiz: 8192 Partial: 2 Poisoning : On Loss : 15264 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 6720 Align : 8 Objects: 20 Tracing : Off Lpadd: 704 Calling shrink now on the cache does nothing because object migration is not enabled (output omitted). If we enable object migration then shrink the cache we expect the object from the second slab to me moved to the first slot in the first slab and the second slab to be removed from the partial list. # echo 'enable' > callfn # slabinfo smo_test --shrink # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 20 ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 1 Sanity Checks : On Total: 8192 SlabObj: 392 Full : 1 Redzoning : On Used : 1120 SlabSiz: 8192 Partial: 0 Poisoning : On Loss : 7072 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 6720 Align : 8 Objects: 20 Tracing : Off Lpadd: 352 We can run the stress tests (with the default number of objects): # cd /sys/kernel/debug/smo # echo 'test' > callfn [ 3.576617] smo: test using nr_objs: 1000 keep: 10 [ 3.580169] smo: Module tests completed successfully Signed-off-by: Tobin C. Harding --- tools/testing/slab/Makefile | 10 + tools/testing/slab/slub_defrag.c | 566 +++++++++++++++++++++++++++++++ 2 files changed, 576 insertions(+) create mode 100644 tools/testing/slab/Makefile create mode 100644 tools/testing/slab/slub_defrag.c diff --git a/tools/testing/slab/Makefile b/tools/testing/slab/Makefile new file mode 100644 index 000000000000..440c2e3e356f --- /dev/null +++ b/tools/testing/slab/Makefile @@ -0,0 +1,10 @@ +obj-m += slub_defrag.o + +KTREE=../../.. + +all: + make -C ${KTREE} M=$(PWD) modules + +clean: + make -C ${KTREE} M=$(PWD) clean + diff --git a/tools/testing/slab/slub_defrag.c b/tools/testing/slab/slub_defrag.c new file mode 100644 index 000000000000..4a5c24394b96 --- /dev/null +++ b/tools/testing/slab/slub_defrag.c @@ -0,0 +1,566 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * SLUB defragmentation a.k.a. Slab Movable Objects (SMO). + * + * This module is used for testing the SLUB allocator. Enables + * userspace to run kernel functions via a debugfs file. + * + * debugfs: /sys/kernel/debugfs/smo/callfn (write only) + * + * String written to `callfn` is parsed by the module and associated + * function is called. See fn_tab for mapping of strings to functions. + */ + +/* debugfs commands accept two optional arguments */ +#define SMO_CMD_DEFAUT_ARG -1 + +#define SMO_DEBUGFS_DIR "smo" +struct dentry *smo_debugfs_root; + +#define SMO_CACHE_NAME "smo_test" +static struct kmem_cache *cachep; + +struct smo_slub_object { + struct list_head list; + char buf[32]; /* Unused except to control size of object */ + long id; +}; + +/* Our list of allocated objects */ +LIST_HEAD(objects); + +static void list_add_to_objects(struct smo_slub_object *so) +{ + /* + * We free from the front of the list so store at the + * tail in order to put holes in the cache when we free. + */ + list_add_tail(&so->list, &objects); +} + +/** + * smo_object_ctor() - SMO object constructor function. + * @ptr: Pointer to memory where the object should be constructed. + */ +void smo_object_ctor(void *ptr) +{ + struct smo_slub_object *so = ptr; + + INIT_LIST_HEAD(&so->list); + memset(so->buf, 0, sizeof(so->buf)); + so->id = -1; +} + +/** + * smo_cache_migrate() - kmem_cache migrate function. + * @cp: kmem_cache pointer. + * @objs: Array of pointers to objects to migrate. + * @size: Number of objects in @objs. + * @node: NUMA node where the object should be allocated. + * @private: Pointer returned by kmem_cache_isolate_func(). + */ +void smo_cache_migrate(struct kmem_cache *cp, void **objs, int size, + int node, void *private) +{ + struct smo_slub_object **so_objs = (struct smo_slub_object **)objs; + struct smo_slub_object *so_old, *so_new; + int i; + + for (i = 0; i < size; i++) { + so_old = so_objs[i]; + + so_new = kmem_cache_alloc_node(cachep, GFP_KERNEL, node); + if (!so_new) { + pr_debug("kmem_cache_alloc failed\n"); + return; + } + + /* Copy object */ + so_new->id = so_old->id; + + /* Update references to old object */ + list_del(&so_old->list); + list_add_to_objects(so_new); + + kmem_cache_free(cachep, so_old); + } +} + +static int smo_enable_cache_mobility(int _unused, int __unused) +{ + /* Enable movable objects: BOOM! */ + kmem_cache_setup_mobility(cachep, NULL, smo_cache_migrate); + pr_info("smo: kmem_cache %s defrag enabled\n", SMO_CACHE_NAME); + return 0; +} + +/* + * smo_alloc_objects() - Allocate objects and store reference. + * @nr_objs: Number of objects to allocate. + * @node: NUMA node to allocate objects on. + * + * Allocates @n smo_slub_objects. Stores a reference to them in + * the global list of objects (at the tail of the list). + * + * Return: The number of objects allocated. + */ +static int smo_alloc_objects(int nr_objs, int node) +{ + struct smo_slub_object *so; + int i; + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1; + if (node == SMO_CMD_DEFAUT_ARG) + node = NUMA_NO_NODE; + + for (i = 0; i < nr_objs; i++) { + if (node == NUMA_NO_NODE) + so = kmem_cache_alloc(cachep, GFP_KERNEL); + else + so = kmem_cache_alloc_node(cachep, GFP_KERNEL, node); + if (!so) { + pr_err("smo: Failed to alloc object %d of %d\n", i, nr_objs); + return i; + } + list_add_to_objects(so); + } + return nr_objs; +} + +/* + * smo_free_object() - Frees n objects from position. + * @nr_objs: Number of objects to free. + * @pos: Position in global list to start freeing. + * + * Iterates over the global list of objects to position @pos then frees @n + * objects from there (or to end of list). Does nothing if @n > list length. + * + * Calling with @n==0 frees all objects starting at @pos. + * + * Return: Number of objects freed. + */ +static int smo_free_object(int nr_objs, int pos) +{ + struct smo_slub_object *cur, *tmp; + int deleted = 0; + int i = 0; + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1; + if (pos == SMO_CMD_DEFAUT_ARG) + pos = 0; + + list_for_each_entry_safe(cur, tmp, &objects, list) { + if (i < pos) { + i++; + continue; + } + + list_del(&cur->list); + kmem_cache_free(cachep, cur); + deleted++; + if (deleted == nr_objs) + break; + } + return deleted; +} + +static int index_for_expected_id(long *expected, int size, long id) +{ + int i; + + /* Array is unsorted, just iterate the whole thing */ + for (i = 0; i < size; i++) { + if (expected[i] == id) + return i; + } + return -1; /* Not found */ +} + +static int assert_have_objects(int nr_objs, int keep) +{ + struct smo_slub_object *cur; + long *expected; /* Array of expected IDs */ + int nr_ids; /* Length of array */ + long id; + int index, i; + + nr_ids = nr_objs / keep + 1; + + expected = kmalloc_array(nr_ids, sizeof(long), GFP_KERNEL); + if (!expected) + return -ENOMEM; + + id = 0; + for (i = 0; i < nr_ids; i++) { + expected[i] = id; + id += keep; + } + + list_for_each_entry(cur, &objects, list) { + index = index_for_expected_id(expected, nr_ids, cur->id); + if (index < 0) { + pr_err("smo: ID not found: %ld\n", cur->id); + return -1; + } + + if (expected[index] == -1) { + pr_err("smo: ID already encountered: %ld\n", cur->id); + return -1; + } + expected[index] = -1; + } + return 0; +} + +/* + * smo_run_module_tests() - Runs unit tests from within the module + * @nr_objs: Number of objects to allocate. + * @keep: Free all but 1 in @keep objects. + * + * Allocates @nr_objects then iterates over the allocated objects + * freeing all but 1 out of every @keep objects i.e. for @keep==10 + * keeps the first object then frees the next 9. + * + * Caller is responsible for ensuring that the cache has at most a + * single slab on the partial list without any objects in it. This is + * easy enough to ensure, just call this when the module is freshly + * loaded. + */ +static int smo_run_module_tests(int nr_objs, int keep) +{ + struct smo_slub_object *so; + struct smo_slub_object *cur, *tmp; + long i; + + if (!list_empty(&objects)) { + pr_err("smo: test requires clean module state\n"); + return -1; + } + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1000; + if (keep == SMO_CMD_DEFAUT_ARG) + keep = 10; + + pr_info("smo: test using nr_objs: %d keep: %d\n", nr_objs, keep); + + /* Perhaps we got called like this 'test 1000' */ + if (keep == 0) { + pr_err("Usage: test \n"); + return -1; + } + + /* Test constructor */ + so = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!so) { + pr_err("smo: Failed to alloc object\n"); + return -1; + } + if (so->id != -1) { + pr_err("smo: Initial state incorrect"); + return -1; + } + kmem_cache_free(cachep, so); + + /* + * Test that object migration is correctly implemented by module + * + * This gives us confidence that if new code correctly enables + * object migration (via correct implementation of migrate and + * isolate functions) then the slub allocator code that does + * object migration is correct. + */ + + for (i = 0; i < nr_objs; i++) { + so = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!so) { + pr_err("smo: Failed to alloc object %ld of %d\n", + i, nr_objs); + return -1; + } + so->id = (long)i; + list_add_to_objects(so); + } + + assert_have_objects(nr_objs, 1); + + i = 0; + list_for_each_entry_safe(cur, tmp, &objects, list) { + if (i++ % keep == 0) + continue; + + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + + /* Verify shrink does nothing when migration is not enabled */ + kmem_cache_shrink(cachep); + assert_have_objects(nr_objs, 1); + + /* Now test shrink */ + kmem_cache_setup_mobility(cachep, NULL, smo_cache_migrate); + kmem_cache_shrink(cachep); + /* + * Because of how migrate function deletes and adds objects to + * the objects list we have no way of knowing the order. We + * want to confirm that we have all the objects after shrink + * that we had before we did the shrink. + */ + assert_have_objects(nr_objs, keep); + + /* cleanup */ + list_for_each_entry_safe(cur, tmp, &objects, list) { + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + kmem_cache_shrink(cachep); /* Remove empty slabs from partial list */ + + pr_info("smo: Module tests completed successfully\n"); + return 0; +} + +/* + * struct functions() - Map command to a function pointer. + */ +struct functions { + char *fn_name; + int (*fn_ptr)(int arg0, int arg1); +} fn_tab[] = { + /* + * Because of the way we parse the function table no command + * may have another command as its prefix. + * i.e. this will break: 'foo' and 'foobar' + */ + {"enable", smo_enable_cache_mobility}, + {"alloc", smo_alloc_objects}, + {"free", smo_free_object}, + {"test", smo_run_module_tests}, +}; + +#define FN_TAB_SIZE (sizeof(fn_tab) / sizeof(struct functions)) + +/* + * parse_cmd_buf() - Gets command and arguments command string. + * @buf: Buffer containing the command string. + * @cmd: Out parameter, pointer to the command. + * @arg1: Out parameter, stores the first argument. + * @arg2: Out parameter, stores the second argument. + * + * Parses and tokenizes the input command buffer. Stores a pointer to the + * command (start of @buf) in @cmd. Stores the converted long values for + * argument 1 and 2 in the respective out parameters @arg1 and @arg2. + * + * Since arguments are optional, if they are not found the default values are + * returned. In order for the caller to differentiate defaults from arguments + * of the same value the number of arguments parsed is returned. + * + * Return: Number of arguments found. + */ +static int parse_cmd_buf(char *buf, char **cmd, long *arg1, long *arg2) +{ + int found; + char *ptr; + int ret; + + *arg1 = SMO_CMD_DEFAUT_ARG; + *arg2 = SMO_CMD_DEFAUT_ARG; + found = 0; + + /* Jump over the command, check if there are any args */ + ptr = strsep(&buf, " "); + if (!ptr || !buf) + return found; + + ptr = strsep(&buf, " "); + ret = kstrtol(ptr, 10, arg1); + if (ret < 0) { + pr_err("failed to convert arg, defaulting to %d. (%s)\n", + SMO_CMD_DEFAUT_ARG, ptr); + return found; + } + found++; + if (!buf) /* No second arg */ + return found; + + ptr = strsep(&buf, " "); + ret = kstrtol(ptr, 10, arg2); + if (ret < 0) { + pr_err("failed to convert arg, defaulting to %d. (%s)\n", + SMO_CMD_DEFAUT_ARG, ptr); + return found; + } + found++; + + return found; +} + +/* + * call_function() - Calls the function described by str. + * @str: ' []' + * + * Does table lookup on , calls appropriate function passing + * as a the argument. Optional arg defaults to 1. + */ +static void call_function(char *str) +{ + char *cmd; + long arg1 = 0; + long arg2 = 0; + int i; + + if (!str) + return; + + (void)parse_cmd_buf(str, &cmd, &arg1, &arg2); + + for (i = 0; i < FN_TAB_SIZE; i++) { + char *fn_name = fn_tab[i].fn_name; + + if (strcmp(fn_name, str) == 0) { + fn_tab[i].fn_ptr(arg1, arg2); + return; /* All done */ + } + } + + pr_err("failed to call function for cmd: %s\n", str); +} + +/* + * smo_callfn_debugfs_write() - debugfs write function. + * @file: User file + * @user_buf: Userspace buffer + * @len: Length of the user space buffer + * @off: Offset within the file + * + * Used for triggering functions by writing command to debugfs file. + * + * echo ' ' > /sys/kernel/debug/smo/callfn + * + * Return: Number of bytes copied if request succeeds, + * the corresponding error code otherwise. + */ +static ssize_t smo_callfn_debugfs_write(struct file *file, + const char __user *ubuf, + size_t len, + loff_t *off) +{ + char *kbuf; + int nbytes = 0; + + if (*off != 0 || len == 0) + return -EINVAL; + + kbuf = kzalloc(len, GFP_KERNEL); + if (!kbuf) + return -ENOMEM; + + nbytes = strncpy_from_user(kbuf, ubuf, len); + if (nbytes < 0) + goto out; + + if (kbuf[nbytes - 1] == '\n') + kbuf[nbytes - 1] = '\0'; + + call_function(kbuf); /* Tokenizes kbuf */ +out: + kfree(kbuf); + return nbytes; +} + +const struct file_operations fops_callfn_debugfs = { + .owner = THIS_MODULE, + .write = smo_callfn_debugfs_write, +}; + +static int __init smo_debugfs_init(void) +{ + struct dentry *d; + + smo_debugfs_root = debugfs_create_dir(SMO_DEBUGFS_DIR, NULL); + d = debugfs_create_file("callfn", 0200, smo_debugfs_root, NULL, + &fops_callfn_debugfs); + if (IS_ERR(d)) + return PTR_ERR(d); + + return 0; +} + +static void __exit smo_debugfs_cleanup(void) +{ + debugfs_remove_recursive(smo_debugfs_root); +} + +static int __init smo_cache_init(void) +{ + cachep = kmem_cache_create(SMO_CACHE_NAME, + sizeof(struct smo_slub_object), + 0, 0, smo_object_ctor); + if (!cachep) + return -1; + + return 0; +} + +static void __exit smo_cache_cleanup(void) +{ + struct smo_slub_object *cur, *tmp; + + list_for_each_entry_safe(cur, tmp, &objects, list) { + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + kmem_cache_destroy(cachep); +} + +static int __init smo_init(void) +{ + int ret; + + ret = smo_cache_init(); + if (ret) { + pr_err("smo: Failed to create cache\n"); + return ret; + } + pr_info("smo: Created kmem_cache: %s\n", SMO_CACHE_NAME); + + ret = smo_debugfs_init(); + if (ret) { + pr_err("smo: Failed to init debugfs\n"); + return ret; + } + pr_info("smo: Created debugfs directory: /sys/kernel/debugfs/%s\n", + SMO_DEBUGFS_DIR); + + pr_info("smo: Test module loaded\n"); + return 0; +} +module_init(smo_init); + +static void __exit smo_exit(void) +{ + smo_debugfs_cleanup(); + smo_cache_cleanup(); + + pr_info("smo: Test module removed\n"); +} +module_exit(smo_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Tobin C. Harding"); +MODULE_DESCRIPTION("SLUB Movable Objects test module."); From patchwork Mon May 20 05:40:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949917 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9FB3E16C1 for ; Mon, 20 May 2019 05:42:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8E92E2864A for ; Mon, 20 May 2019 05:42:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 81E7B2863D; Mon, 20 May 2019 05:42:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 784262863D for ; Mon, 20 May 2019 05:42:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 44AF96B0010; Mon, 20 May 2019 01:42:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3D43D6B0266; Mon, 20 May 2019 01:42:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29D186B0269; Mon, 20 May 2019 01:42:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 09AF56B0010 for ; Mon, 20 May 2019 01:42:11 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id w41so13199678qth.20 for ; Sun, 19 May 2019 22:42:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=BGctWGdyOjvDA8OBilWH9oV/RXRD9aYUdmguPRAzHqNoZ360eq21bCjKNRCCEJadXu v2gk56uh8+ivSfQ7rTeQdgK0mJ3QrrqOG+bLPhYKzE0wV/Wde+DF8H7irD5fxjiV4Tpx g/bbhlYYxrMReFJVVd1vL4AMgdPc482fmePxRD4XYd0emNqAOIHoDRXRCBv7O9bwnY44 9Hq+lBA9Zwpk9bhflL6r8m61Tz3+E9jJxz7bpHacubj1ao+NJjdBaAnkGjwxq9EaX0Fh CoESIxZeLdg4ULH4pWCfp4By56d9Y8lRx7jkVO4mMLNLXA3ebjoQRbL5x0Kb+ftrKI65 3ZXQ== X-Gm-Message-State: APjAAAURdiZdd7vnAP08HGYDOMm7BIOtIaNdF8wuSM2odu38riOjcR5+ kMf2li+boXIrUhp9+lr0K5EdZnw/iczXCavzAqkm7FdElPeTkUeNFbyfyNOnf/gor5tGgRVf0lm TOQcsw+9ArVOsbfJVDCho3eKoSTYU4DDDhhcjlbqRUeIsyFcxEoxgIl5yBO5L6o0= X-Received: by 2002:a0c:878e:: with SMTP id 14mr60455456qvj.103.1558330930736; Sun, 19 May 2019 22:42:10 -0700 (PDT) X-Google-Smtp-Source: APXvYqzrzEakahZ1ik79eFMpxoujeaPXaChgSAppNuwwVbiL5aDiRCQHKNQ+fvt5wepXmWUgLnI6 X-Received: by 2002:a0c:878e:: with SMTP id 14mr60455366qvj.103.1558330928857; Sun, 19 May 2019 22:42:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330928; cv=none; d=google.com; s=arc-20160816; b=nfOkIblLPYl1CD9LlGgschQQdzyVm/Jk4Plgy8x354etbqz6QOr959Fbf6hSdBl4+H ngVTZd2CpXMF2yztgc9gX5Aogj94gctDT5TdfLZSqBYrhTZUpUdFCicSUnez/9UqtN73 LGpmZccktw2q6mli3Wq3gI96knFqjGOuHnk1t6lcFj9tcXaDY+3DMrqCjrWpx+4YfC5i GXuODky2cyq8ocVjyDEBMEqh1h3NNmA99ZAdruhJrIXRMypuYpVn5kPr6zbMuwJs9iuj Pf+XtcJXecAmzrPrAa/FNLhqpPLTis4rNi5hxg9rtn7R03kfBeiG3QVA4e13y4f+AUTX SIAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=FDWuG87plXM7l4t+NDFPg2SqGvERKkmYUNYxINT+xDIUIIRqTvkksOW1HvabL0SWXC dDHG1Izt9Nxsg516eZdEQvT3z9AH22pCWOf22Pwirv618PUdx63kJ0jmklVc9gG2VdBE JKL+LOyUt0xfK5BpjrUunl/Y5vuqzTEW3K+Mh6ESlkLN4oRDNaSeq/WJMiKnxfP/UtFE RVGUzs39pUEW36UGUvpiW8phBAgX3dBmpichB4pF6sBu9E0EdLHN5P3mBnjJ5EKc85UX boYM1SZfCv4I33vJKj+5Lfnfz+zmE5WH1i/MVcpw6eAHvTVpDPyCtKj0zNjqFzeMeYxV 6YNA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=waZHIrf9; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id c8si8356811qvc.154.2019.05.19.22.42.08 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:42:08 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=waZHIrf9; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 91A2D9D8C; Mon, 20 May 2019 01:42:08 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:42:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=waZHIrf9 TbDMMRcU23/1TllYoJPkCaDUFu05KKtooyOJsX46KcnUh19F/s0xpoXJUmVGgMQK KGXlbkpGYOEA7ZVrqcAxoePZ8epz9Ls6skj3WNZQ1hiiTlz04gjCTl4Mbt6/YSFQ l2LtAqm6Xb76ZP88h6sN5htsKwtxbgSCxCYSStSMODSAqVTLVAkXl0WxvCq2cyan /IWC9bfpGKPpgW9YxNSsvVM/wP6/CT8Xaowurca0EC4uvLLyJa+FWubG0k3hXQE2 aoq2Nx0BG7k2FcvvNvQM7q5oj8LU5xzeRR63npfwvaqEBAnx9OyxG0MsGQ4VXY7S pTXeqgGNAB2mEQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepje X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 66C8E80064; Mon, 20 May 2019 01:42:01 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 08/16] tools/testing/slab: Add object migration test suite Date: Mon, 20 May 2019 15:40:09 +1000 Message-Id: <20190520054017.32299-9-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just added a module that enables testing the SLUB allocators ability to defrag/shrink caches via movable objects. Tests are better when they are automated. Add automated testing via a python script for SLUB movable objects. Example output: $ cd path/to/linux/tools/testing/slab $ /slub_defrag.py Please run script as root $ sudo ./slub_defrag.py $ sudo ./slub_defrag.py --debug Loading module ... Slab cache smo_test created Objects per slab: 20 Running sanity checks ... Running module stress test (see dmesg for additional test output) ... Removing module slub_defrag ... Loading module ... Slab cache smo_test created Running test non-movable ... testing slab 'smo_test' prior to enabling movable objects ... verified non-movable slabs are NOT shrinkable Running test movable ... testing slab 'smo_test' after enabling movable objects ... verified movable slabs are shrinkable Removing module slub_defrag ... Signed-off-by: Tobin C. Harding --- tools/testing/slab/slub_defrag.c | 1 + tools/testing/slab/slub_defrag.py | 451 ++++++++++++++++++++++++++++++ 2 files changed, 452 insertions(+) create mode 100755 tools/testing/slab/slub_defrag.py diff --git a/tools/testing/slab/slub_defrag.c b/tools/testing/slab/slub_defrag.c index 4a5c24394b96..8332e69ee868 100644 --- a/tools/testing/slab/slub_defrag.c +++ b/tools/testing/slab/slub_defrag.c @@ -337,6 +337,7 @@ static int smo_run_module_tests(int nr_objs, int keep) /* * struct functions() - Map command to a function pointer. + * If you update this please update the documentation in slub_defrag.py */ struct functions { char *fn_name; diff --git a/tools/testing/slab/slub_defrag.py b/tools/testing/slab/slub_defrag.py new file mode 100755 index 000000000000..41747c0db39b --- /dev/null +++ b/tools/testing/slab/slub_defrag.py @@ -0,0 +1,451 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import subprocess +import sys +from os import path + +# SLUB Movable Objects test suite. +# +# Requirements: +# - CONFIG_SLUB=y +# - CONFIG_SLUB_DEBUG=y +# - The slub_defrag module in this directory. + +# Test SMO using a kernel module that enables triggering arbitrary +# kernel code from userspace via a debugfs file. +# +# Module code is in ./slub_defrag.c, basically the functionality is as +# follows: +# +# - Creates debugfs file /sys/kernel/debugfs/smo/callfn +# - Writes to 'callfn' are parsed as a command string and the function +# associated with command is called. +# - Defines 4 commands (all commands operate on smo_test cache): +# - 'test': Runs module stress tests. +# - 'alloc N': Allocates N slub objects +# - 'free N POS': Frees N objects starting at POS (see below) +# - 'enable': Enables SLUB Movable Objects +# +# The module maintains a list of allocated objects. Allocation adds +# objects to the tail of the list. Free'ing frees from the head of the +# list. This has the effect of creating free slots in the slab. For +# finer grained control over where in the cache slots are free'd POS +# (position) argument may be used. + +# The main() function is reasonably readable; the test suite does the +# following: +# +# 1. Runs the module stress tests. +# 2. Tests the cache without movable objects enabled. +# - Creates multiple partial slabs as explained above. +# - Verifies that partial slabs are _not_ removed by shrink (see below). +# 3. Tests the cache with movable objects enabled. +# - Creates multiple partial slabs as explained above. +# - Verifies that partial slabs _are_ removed by shrink (see below). + +# The sysfs file /sys/kernel/slab//shrink enables calling the +# function kmem_cache_shrink() (see mm/slab_common.c and mm/slub.cc). +# Shrinking a cache attempts to consolidate all partial slabs by moving +# objects if object migration is enable for the cache, otherwise +# shrinking a cache simply re-orders the partial list so as most densely +# populated slab are at the head of the list. + +# Enable/disable debugging output (also enabled via -d | --debug). +debug = False + +# Used in debug messages and when running `insmod`. +MODULE_NAME = "slub_defrag" + +# Slab cache created by the test module. +CACHE_NAME = "smo_test" + +# Set by get_slab_config() +objects_per_slab = 0 +pages_per_slab = 0 +debugfs_mounted = False # Set to true if we mount debugfs. + + +def eprint(*args, **kwargs): + print(*args, file=sys.stderr, **kwargs) + + +def dprint(*args, **kwargs): + if debug: + print(*args, file=sys.stderr, **kwargs) + + +def run_shell(cmd): + return subprocess.call([cmd], shell=True) + + +def run_shell_get_stdout(cmd): + return subprocess.check_output([cmd], shell=True) + + +def assert_root(): + user = run_shell_get_stdout('whoami') + if user != b'root\n': + eprint("Please run script as root") + sys.exit(1) + + +def mount_debugfs(): + mounted = False + + # Check if debugfs is mounted at a known mount point. + ret = run_shell('mount -l | grep /sys/kernel/debug > /dev/null 2>&1') + if ret != 0: + run_shell('mount -t debugfs none /sys/kernel/debug/') + mounted = True + dprint("Mounted debugfs on /sys/kernel/debug") + + return mounted + + +def umount_debugfs(): + dprint("Un-mounting debugfs") + run_shell('umount /sys/kernel/debug') + + +def load_module(): + """Loads the test module. + + We need a clean slab state to start with so module must + be loaded by the test suite. + """ + ret = run_shell('lsmod | grep %s > /dev/null' % MODULE_NAME) + if ret == 0: + eprint("Please unload slub_defrag module before running test suite") + return -1 + + dprint('Loading module ...') + ret = run_shell('insmod %s.ko' % MODULE_NAME) + if ret != 0: # ret==1 on error + return -1 + + dprint("Slab cache %s created" % CACHE_NAME) + return 0 + + +def unload_module(): + ret = run_shell('lsmod | grep %s > /dev/null' % MODULE_NAME) + if ret == 0: + dprint('Removing module %s ...' % MODULE_NAME) + run_shell('rmmod %s > /dev/null 2>&1' % MODULE_NAME) + + +def get_sysfs_value(filename): + """ + Parse slab sysfs files (single line: '20 N0=20') + """ + path = '/sys/kernel/slab/smo_test/%s' % filename + f = open(path, "r") + s = f.readline() + tokens = s.split(" ") + + return int(tokens[0]) + + +def get_nr_objects_active(): + return get_sysfs_value('objects') + + +def get_nr_objects_total(): + return get_sysfs_value('total_objects') + + +def get_nr_slabs_total(): + return get_sysfs_value('slabs') + + +def get_nr_slabs_partial(): + return get_sysfs_value('partial') + + +def get_nr_slabs_full(): + return get_nr_slabs_total() - get_nr_slabs_partial() + + +def get_slab_config(): + """Get relevant information from sysfs.""" + global objects_per_slab + + objects_per_slab = get_sysfs_value('objs_per_slab') + if objects_per_slab < 0: + return -1 + + dprint("Objects per slab: %d" % objects_per_slab) + return 0 + + +def verify_state(nr_objects_active, nr_objects_total, + nr_slabs_partial, nr_slabs_full, nr_slabs_total, msg=''): + err = 0 + got_nr_objects_active = get_nr_objects_active() + got_nr_objects_total = get_nr_objects_total() + got_nr_slabs_partial = get_nr_slabs_partial() + got_nr_slabs_full = get_nr_slabs_full() + got_nr_slabs_total = get_nr_slabs_total() + + if got_nr_objects_active != nr_objects_active: + err = -1 + + if got_nr_objects_total != nr_objects_total: + err = -2 + + if got_nr_slabs_partial != nr_slabs_partial: + err = -3 + + if got_nr_slabs_full != nr_slabs_full: + err = -4 + + if got_nr_slabs_total != nr_slabs_total: + err = -5 + + if err != 0: + dprint("Verify state: %s" % msg) + dprint(" what\t\t\twant\tgot") + dprint("-----------------------------------------") + dprint(" %s\t%d\t%d" % ('nr_objects_active', nr_objects_active, got_nr_objects_active)) + dprint(" %s\t%d\t%d" % ('nr_objects_total', nr_objects_total, got_nr_objects_total)) + dprint(" %s\t%d\t%d" % ('nr_slabs_partial', nr_slabs_partial, got_nr_slabs_partial)) + dprint(" %s\t\t%d\t%d" % ('nr_slabs_full', nr_slabs_full, got_nr_slabs_full)) + dprint(" %s\t%d\t%d\n" % ('nr_slabs_total', nr_slabs_total, got_nr_slabs_total)) + + return err + + +def exec_via_sysfs(command): + ret = run_shell('echo %s > /sys/kernel/debug/smo/callfn' % command) + if ret != 0: + eprint("Failed to echo command to sysfs: %s" % command) + + return ret + + +def enable_movable_objects(): + return exec_via_sysfs('enable') + + +def alloc(n): + exec_via_sysfs("alloc %d" % n) + + +def free(n, pos = 0): + exec_via_sysfs('free %d %d' % (n, pos)) + + +def shrink(): + ret = run_shell('slabinfo smo_test -s') + if ret != 0: + eprint("Failed to execute slabinfo -s") + + +def sanity_checks(): + # Verify everything is 0 to start with. + return verify_state(0, 0, 0, 0, 0, "sanity check") + + +def test_non_movable(): + one_over = objects_per_slab + 1 + + dprint("testing slab 'smo_test' prior to enabling movable objects ...") + + alloc(one_over) + + objects_active = one_over + objects_total = objects_per_slab * 2 + slabs_partial = 1 + slabs_full = 1 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: initial allocation") + if ret != 0: + eprint("test_non_movable: failed to verify initial state") + return -1 + + # Free object from first slot of first slab. + free(1) + objects_active = one_over - 1 + objects_total = objects_per_slab * 2 + slabs_partial = 2 + slabs_full = 0 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: after free") + if ret != 0: + eprint("test_non_movable: failed to verify after free") + return -1 + + # Non-movable cache, shrink should have no effect. + shrink() + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: after shrink") + if ret != 0: + eprint("test_non_movable: failed to verify after shrink") + return -1 + + # Cleanup + free(objects_per_slab) + shrink() + + dprint("verified non-movable slabs are NOT shrinkable") + return 0 + + +def test_movable(): + one_over = objects_per_slab + 1 + + dprint("testing slab 'smo_test' after enabling movable objects ...") + + alloc(one_over) + + objects_active = one_over + objects_total = objects_per_slab * 2 + slabs_partial = 1 + slabs_full = 1 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: initial allocation") + if ret != 0: + eprint("test_movable: failed to verify initial state") + return -1 + + # Free object from first slot of first slab. + free(1) + objects_active = one_over - 1 + objects_total = objects_per_slab * 2 + slabs_partial = 2 + slabs_full = 0 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: after free") + if ret != 0: + eprint("test_movable: failed to verify after free") + return -1 + + # movable cache, shrink should move objects and free slab. + shrink() + objects_active = one_over - 1 + objects_total = objects_per_slab * 1 + slabs_partial = 0 + slabs_full = 1 + slabs_total = 1 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: after shrink") + if ret != 0: + eprint("test_movable: failed to verify after shrink") + return -1 + + # Cleanup + free(objects_per_slab) + shrink() + + dprint("verified movable slabs are shrinkable") + return 0 + + +def dprint_start_test(test): + dprint("Running %s ..." % test) + + +def dprint_done(): + dprint("") + + +def run_test(fn, desc): + dprint_start_test(desc) + ret = fn() + if ret < 0: + fail_test(desc) + dprint_done() + + +# Load and unload the module for this test to ensure clean state. +def run_module_stress_test(): + dprint("Running module stress test (see dmesg for additional test output) ...") + + unload_module() + ret = load_module() + if ret < 0: + cleanup_and_exit(ret) + + exec_via_sysfs("test"); + + unload_module() + + dprint() + + +def fail_test(msg): + eprint("\nFAIL: test failed: '%s' ... aborting\n" % msg) + cleanup_and_exit(1) + + +def display_help(): + print("Usage: %s [OPTIONS]\n" % path.basename(sys.argv[0])) + print("\tRuns defrag test suite (a.k.a. SLUB Movable Objects)\n") + print("OPTIONS:") + print("\t-d | --debug Enable verbose debug output") + print("\t-h | --help Print this help and exit") + + +def cleanup_and_exit(return_code): + global debugfs_mounted + + if debugfs_mounted == True: + umount_debugfs() + + unload_module() + + sys.exit(return_code) + + +def main(): + global debug + + if len(sys.argv) > 1: + if sys.argv[1] == '-h' or sys.argv[1] == '--help': + display_help() + sys.exit(0) + + if sys.argv[1] == '-d' or sys.argv[1] == '--debug': + debug = True + + assert_root() + + # Use cleanup_and_exit() instead of sys.exit() after mounting debugfs. + debugfs_mounted = mount_debugfs() + + # Loads and unloads the module. + run_module_stress_test() + + ret = load_module() + if (ret < 0): + cleanup_and_exit(ret) + + ret = get_slab_config() + if (ret != 0): + fail_test("get slab config details") + + run_test(sanity_checks, "sanity checks") + + run_test(test_non_movable, "test non-movable") + + ret = enable_movable_objects() + if (ret != 0): + fail_test("enable movable objects") + + run_test(test_movable, "test movable") + + cleanup_and_exit(0) + +if __name__== "__main__": + main() From patchwork Mon May 20 05:40:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949919 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E07E1390 for ; Mon, 20 May 2019 05:42:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0CEBC2863D for ; Mon, 20 May 2019 05:42:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 00D3628644; Mon, 20 May 2019 05:42:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 325D2285ED for ; Mon, 20 May 2019 05:42:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0EED56B0266; Mon, 20 May 2019 01:42:18 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 09FBD6B0269; Mon, 20 May 2019 01:42:18 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EAA486B026A; Mon, 20 May 2019 01:42:17 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id CD6486B0266 for ; Mon, 20 May 2019 01:42:17 -0400 (EDT) Received: by mail-qt1-f199.google.com with SMTP id w34so13200138qtc.16 for ; Sun, 19 May 2019 22:42:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=F5aR/6PSAAo/4gqYLlgL9w8L/omD4wgEh0XNiCKl+dI=; b=ZcBQOLFCdcg3jFVGIo1qFgn/JLd36szFtga5iJ2jnz4TBFB9lh3c6zLiQB7hDuarKi CNla9Aas7WHndT7MjZzlM0fqde4W+6o2LdgBObsyGGY96xhwyFCjWrQ62usAm4GYeEiE CtHav9b57Ka7vhNibWvqfq0lo1gqPzvB2Uco2Tod7m2toJ1UMpiY21Hl6vxiZRRTknqy cwsgJuVjgDdJ0peoOeKy3zkTPygMXfKcLbUU/QvShxl7WwFrjEeF3rkV/ucIDUMj4/YB WHsBM979K9K4Uj0Mvho/QilLplU6d0zZ8lzncuVRNldnrGh2V6WjgdefUxnm8PonB/wr hpFw== X-Gm-Message-State: APjAAAU0ZfbhhKp9jhXBG8DLJE6GR6i+LxuvAS/47JHtaNBr46jl2KDf mOEfDbgxxGL++CNDHDWwbijHu9S1vtWYF/PTrMdkh0PTzXM5w2aIij73OgIfIHOBrhbJhzVTtrJ 7I+/y7VKMD0EQvt4oOYRpWQFoc2OeCFenEEWbiV4GgwsL3lI1QakuTXUjkuu+2X8= X-Received: by 2002:a05:620a:1346:: with SMTP id c6mr56223389qkl.275.1558330937517; Sun, 19 May 2019 22:42:17 -0700 (PDT) X-Google-Smtp-Source: APXvYqwOYq6fz0eWrWE2vuLDbp4ILFMK4YqFw1yWfPwT5BMdMGZiRavF5Yk44GML5/2uvQnks1Wn X-Received: by 2002:a05:620a:1346:: with SMTP id c6mr56223326qkl.275.1558330936213; Sun, 19 May 2019 22:42:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330936; cv=none; d=google.com; s=arc-20160816; b=vF5fTnN/3iFvhQ5dZcWO59vn34exYivb89uG54LSiq4PPwzazIvSaJ/kw6cj+y+OCD YBN0J9fUJniWy2C+fRNSswj+ErdG4nlEOuwfPZ2mTay4vc5hVYJi3ALRvuMBwq/DMB7/ Mp+oC6dLi7tECsxU/jCazx2NdZ90z7ihn7DdmWTSriX4/+dczLed7yjj6M5ryS9umbkZ nuRrDdwgIuvomaQr85Ug3xixyZeBOwlAVbYQhSWpuiERwkrnTUJvc0/EwPMulbcVrMrJ ByVAG0Fmk9MDgsbyaN+lWUQLDe7TH9NPm28f47U7f6kiTRryuI3Go2/iUHAV3uemWsLe ZhSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=F5aR/6PSAAo/4gqYLlgL9w8L/omD4wgEh0XNiCKl+dI=; b=DR0/r2XA2f634SKT1CVrWSG9U27d6SoA/QyhM9yzvEmnVMmlYugFD65IJpzMbnUG0r GtV5zgpiG3mLDk06rodta2hpjsUSyS/qETIC+oq9oAEpMYNQcKGUy0u/ADuGE2P2WeIL C8jqNpnwuH+poVoaT4SwzvfregAHXbr1bIkU6N6DyPekWdfoPg9/7d/MQaw1qLiP06g0 ZwTWjXyVD5nYPiv01Jy1lzlwWAJquqzXBDsa9aL0NJIUb5hqh0Wmk9Fnf3kUbk5kNhXx x/PXgejK/q1rqUM3cVDOBC6lWSKwI4ET6wk/yTSgRMNBnzFa7IeGY2pIe/+rQTGC+Xtz kr9g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=4q0hWeJv; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id u20si1277192qka.162.2019.05.19.22.42.16 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:42:16 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=4q0hWeJv; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id C619AA829; Mon, 20 May 2019 01:42:15 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:42:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=F5aR/6PSAAo/4gqYLlgL9w8L/omD4wgEh0XNiCKl+dI=; b=4q0hWeJv FFIFSzB6p1Y6Ld4qHXbTythAidiWhdLsKMnKSWiR5p01cWZpv4hFaNHhAkwteTso YBCvHKCyNhdOhs2m6HTXibD3ARgYfxV6x1grE5JpOEOtTAtRRWn4gkw1YPFIiBJ3 scfIfzv1LsLBBxCWhjaIkoOEtBTqGcdTgxy+taAh5a9gAIWMiL84hsZ/l2RTKUBE bM/JxmtQ79/pYZrYpJ/hRSYqVPf4sG4307TJJzZmOD4rooSaBremXmbKUkJqkve8 8vsWXXbVvc4fWaeKbUeGFaf+oyAb4nqgZ00zTN+NqAs5xZ8/PwgUyD2yUM0BDRJZ hPdz6VR3FKdrNA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepke X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 9B96980061; Mon, 20 May 2019 01:42:08 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 09/16] lib: Separate radix_tree_node and xa_node slab cache Date: Mon, 20 May 2019 15:40:10 +1000 Message-Id: <20190520054017.32299-10-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Earlier, Slab Movable Objects (SMO) was implemented. The XArray is now able to take advantage of SMO in order to make xarray nodes movable (when using the SLUB allocator). Currently the radix tree uses the same slab cache as the XArray. Only XArray nodes are movable _not_ radix tree nodes. We can give the radix tree its own slab cache to overcome this. In preparation for implementing XArray object migration (xa_node objects) via Slab Movable Objects add a slab cache solely for XArray nodes and make the XArray use this slab cache instead of the radix_tree_node slab cache. Cc: Matthew Wilcox Signed-off-by: Tobin C. Harding --- include/linux/xarray.h | 3 +++ init/main.c | 2 ++ lib/radix-tree.c | 2 +- lib/xarray.c | 48 ++++++++++++++++++++++++++++++++++-------- 4 files changed, 45 insertions(+), 10 deletions(-) diff --git a/include/linux/xarray.h b/include/linux/xarray.h index 0e01e6129145..773f91f8e1db 100644 --- a/include/linux/xarray.h +++ b/include/linux/xarray.h @@ -42,6 +42,9 @@ #define BITS_PER_XA_VALUE (BITS_PER_LONG - 1) +/* Called from init/main.c */ +void xarray_slabcache_init(void); + /** * xa_mk_value() - Create an XArray entry from an integer. * @v: Value to store in XArray. diff --git a/init/main.c b/init/main.c index 5a2c69b4d7b3..e89915ffbe26 100644 --- a/init/main.c +++ b/init/main.c @@ -106,6 +106,7 @@ static int kernel_init(void *); extern void init_IRQ(void); extern void radix_tree_init(void); +extern void xarray_slabcache_init(void); /* * Debug helper: via this flag we know that we are in 'early bootup code' @@ -621,6 +622,7 @@ asmlinkage __visible void __init start_kernel(void) "Interrupts were enabled *very* early, fixing it\n")) local_irq_disable(); radix_tree_init(); + xarray_slabcache_init(); /* * Set up housekeeping before setting up workqueues to allow the unbound diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 14d51548bea6..edbfb530ba73 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -44,7 +44,7 @@ /* * Radix tree node cache. */ -struct kmem_cache *radix_tree_node_cachep; +static struct kmem_cache *radix_tree_node_cachep; /* * The radix tree is variable-height, so an insert operation not only has diff --git a/lib/xarray.c b/lib/xarray.c index 6be3acbb861f..a528a5277c9d 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -27,6 +27,8 @@ * @entry refers to something stored in a slot in the xarray */ +static struct kmem_cache *xa_node_cachep; + static inline unsigned int xa_lock_type(const struct xarray *xa) { return (__force unsigned int)xa->xa_flags & 3; @@ -244,9 +246,21 @@ void *xas_load(struct xa_state *xas) } EXPORT_SYMBOL_GPL(xas_load); -/* Move the radix tree node cache here */ -extern struct kmem_cache *radix_tree_node_cachep; -extern void radix_tree_node_rcu_free(struct rcu_head *head); +void xa_node_rcu_free(struct rcu_head *head) +{ + struct xa_node *node = container_of(head, struct xa_node, rcu_head); + + /* + * Must only free zeroed nodes into the slab. We can be left with + * non-NULL entries by radix_tree_free_nodes, so clear the entries + * and tags here. + */ + memset(node->slots, 0, sizeof(node->slots)); + memset(node->tags, 0, sizeof(node->tags)); + INIT_LIST_HEAD(&node->private_list); + + kmem_cache_free(xa_node_cachep, node); +} #define XA_RCU_FREE ((struct xarray *)1) @@ -254,7 +268,7 @@ static void xa_node_free(struct xa_node *node) { XA_NODE_BUG_ON(node, !list_empty(&node->private_list)); node->array = XA_RCU_FREE; - call_rcu(&node->rcu_head, radix_tree_node_rcu_free); + call_rcu(&node->rcu_head, xa_node_rcu_free); } /* @@ -270,7 +284,7 @@ static void xas_destroy(struct xa_state *xas) if (!node) return; XA_NODE_BUG_ON(node, !list_empty(&node->private_list)); - kmem_cache_free(radix_tree_node_cachep, node); + kmem_cache_free(xa_node_cachep, node); xas->xa_alloc = NULL; } @@ -298,7 +312,7 @@ bool xas_nomem(struct xa_state *xas, gfp_t gfp) xas_destroy(xas); return false; } - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc(xa_node_cachep, gfp); if (!xas->xa_alloc) return false; XA_NODE_BUG_ON(xas->xa_alloc, !list_empty(&xas->xa_alloc->private_list)); @@ -327,10 +341,10 @@ static bool __xas_nomem(struct xa_state *xas, gfp_t gfp) } if (gfpflags_allow_blocking(gfp)) { xas_unlock_type(xas, lock_type); - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc(xa_node_cachep, gfp); xas_lock_type(xas, lock_type); } else { - xas->xa_alloc = kmem_cache_alloc(radix_tree_node_cachep, gfp); + xas->xa_alloc = kmem_cache_alloc(xa_node_cachep, gfp); } if (!xas->xa_alloc) return false; @@ -358,7 +372,7 @@ static void *xas_alloc(struct xa_state *xas, unsigned int shift) if (node) { xas->xa_alloc = NULL; } else { - node = kmem_cache_alloc(radix_tree_node_cachep, + node = kmem_cache_alloc(xa_node_cachep, GFP_NOWAIT | __GFP_NOWARN); if (!node) { xas_set_err(xas, -ENOMEM); @@ -1971,6 +1985,22 @@ void xa_destroy(struct xarray *xa) } EXPORT_SYMBOL(xa_destroy); +static void xa_node_ctor(void *arg) +{ + struct xa_node *node = arg; + + memset(node, 0, sizeof(*node)); + INIT_LIST_HEAD(&node->private_list); +} + +void __init xarray_slabcache_init(void) +{ + xa_node_cachep = kmem_cache_create("xarray_node", + sizeof(struct xa_node), 0, + SLAB_PANIC | SLAB_RECLAIM_ACCOUNT, + xa_node_ctor); +} + #ifdef XA_DEBUG void xa_dump_node(const struct xa_node *node) { From patchwork Mon May 20 05:40:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949923 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7D57A1390 for ; Mon, 20 May 2019 05:42:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C2AB282EC for ; Mon, 20 May 2019 05:42:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6013228649; Mon, 20 May 2019 05:42:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D7EEE28646 for ; Mon, 20 May 2019 05:42:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B767F6B0269; Mon, 20 May 2019 01:42:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B263E6B026A; Mon, 20 May 2019 01:42:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A159B6B026B; Mon, 20 May 2019 01:42:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 836286B0269 for ; Mon, 20 May 2019 01:42:24 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id b46so13233652qte.6 for ; Sun, 19 May 2019 22:42:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=US5HWl6Yr+o7Tqlb3g83eU3pKx3YxwNaogU4TX1LtRs=; b=Bar/2Ns0A9o5SW3o/3Z83fI6RERJV45TFObZL2eHbqHfwRv+oYimfyTElLwVDd/8yy kD2k6r2/6yOuv9pCSZJbU0k5+BaPo3N/TVq4r5AyI508VUixjiOuWTDZNIE5P9K4V6l1 uUqMk+yLe/k369uYum1ZADczT5rSDTQ353NGZrNaT3jIydCBmYIp3isfn/bmBjPJ0Yyj 7v9UpfvEAI4HeDeiMqos898bDyOJheFcctGNWTrEULmU84sepsHb0qi8a2SuvIbRO4tI TOdgkO1iP+LyKuRQeWEoA3pxRJ3br1ILGyS1ZwK5QP5Kt9aXkXxpsAvJIHiQfgnzjI+6 IiIA== X-Gm-Message-State: APjAAAWoFnBLYEWkKx/3h+RAokf5fI6GczW97B5ugnnZexJnLpZDwEvQ aIWoOF/ynypILzr+Eu94VyAQ/SOcfI0ddJBJlGJEIgzCIgEsZiWobp0iZWvbBuktGmuBn2xW2Th 6ZwCHOooM+f5D1G0/bVe9+imstlJbS5qtm4EjSENRVstj3ALquh0AW9HBamKv0Ow= X-Received: by 2002:a0c:98ab:: with SMTP id f40mr50984249qvd.177.1558330944302; Sun, 19 May 2019 22:42:24 -0700 (PDT) X-Google-Smtp-Source: APXvYqwK20FjSDoudI30XPA0naBBiCs9xA/SkhF7QzQlv4Aq6L/bHuHJqfcc0oOblPh5toHTcoae X-Received: by 2002:a0c:98ab:: with SMTP id f40mr50984197qvd.177.1558330943297; Sun, 19 May 2019 22:42:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330943; cv=none; d=google.com; s=arc-20160816; b=hhWvvzosXEmSejY3Z8H+xJjvr+5RfWgdn2ZlP1h3cYl4rCjuWXhA+icoWY4LGm3G6G PEsfym2JwA+lJ/y++osO1OeJrLENQtlWAXjXcsXOVvUJRBTaBdIsnNyDq3vzYJ/lHjXQ w5i1ftDrfiHXjJb8uR++sF6mO4FN8eFTpie72hFvMwW2fplAMGxpWwTY5127Hszr9/Bt jjN8jczRN1k4CYr3JX8BTDV0r7qBGeYTSTn/pkZZKzVdX6zlxPTrujS2BY1j0cH0sXrf 8peD/B+qrZ4JmYeyBO3r240vS96+mPUHmU0HPAbrGyO5PwySFC2UbdFfaP1ytdfx9opK OfGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=US5HWl6Yr+o7Tqlb3g83eU3pKx3YxwNaogU4TX1LtRs=; b=qFEH7Ln093Drap+F/e7qOxIPSTI0cdlPdG24xtkaUVXUkyDOO0crS63OBzTB3NClgI zyIGHd+fVRy39gtbdzzEa3w/3uU3CefdfEdfDBNErDFWbZFFdJCbTdgz1CLueU+thjy1 h3py3XJ8i3gDEeRWUrYkZnyvfy/21h4RY7+AoTVU9ZwdZsuYesaaaH6KMlwBvgnPoRBl +qnXaowdgORr4PveLOCWunUeHBaC3xllg2R28ueLUDUyrNQm/XPhHX6DW8PB43tT98cH PiCRQuttHluHsoaJSRNeElHbK/Fx1kWcyJqTkweLIVUEwqmOsn0TyvSc5rToDWrVmHQa 2kxw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=kc0kWlJs; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id s92si3604756qtd.48.2019.05.19.22.42.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:42:23 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=kc0kWlJs; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 045C9BC13; Mon, 20 May 2019 01:42:23 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:42:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=US5HWl6Yr+o7Tqlb3g83eU3pKx3YxwNaogU4TX1LtRs=; b=kc0kWlJs 0NmjnfFnQi6LJk07eViEaoQIS4EmoXA3ywM5OjQF4mkY8O/fnqPxMiPO4eJ8GufP 9CEETvaMZ5OeCvk7DNGRQS4MdzhgMBOx9/f6mwIBOaRjjW0gsGAba7cI/Vj+rkCH dqdNfbqvrGDv5OgqZmFlGxGS90D2eESO0qDO3VN37SwQcI0YUHkqRMeyUQEyyUA4 1vho7FU/MKIdoJyvgGcNb7BTfBvEDaPlR6ZpSUWpGA1Iqi60m33B1iIIBIoF2fNW xxp75W69/QaOZpEOP/JEVcwmXEcp0qnKw9YflyGEDCy0BKnRld5h5QLLZnCSC9DH RN4TnjdXHArysw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeple X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id DBBF380064; Mon, 20 May 2019 01:42:15 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 10/16] xarray: Implement migration function for xa_node objects Date: Mon, 20 May 2019 15:40:11 +1000 Message-Id: <20190520054017.32299-11-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Recently Slab Movable Objects (SMO) was implemented for the SLUB allocator. The XArray can take advantage of this and make the xa_node slab cache objects movable. Implement functions to migrate objects and activate SMO when we initialise the XArray slab cache. This is based on initial code by Matthew Wilcox and was modified to work with slab object migration. Cc: Matthew Wilcox Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- lib/xarray.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+) diff --git a/lib/xarray.c b/lib/xarray.c index a528a5277c9d..c6b077f59e88 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1993,12 +1993,73 @@ static void xa_node_ctor(void *arg) INIT_LIST_HEAD(&node->private_list); } +static void xa_object_migrate(struct xa_node *node, int numa_node) +{ + struct xarray *xa = READ_ONCE(node->array); + void __rcu **slot; + struct xa_node *new_node; + int i; + + /* Freed or not yet in tree then skip */ + if (!xa || xa == XA_RCU_FREE) + return; + + new_node = kmem_cache_alloc_node(xa_node_cachep, GFP_KERNEL, numa_node); + if (!new_node) { + pr_err("%s: slab cache allocation failed\n", __func__); + return; + } + + xa_lock_irq(xa); + + /* Check again..... */ + if (xa != node->array) { + node = new_node; + goto unlock; + } + + memcpy(new_node, node, sizeof(struct xa_node)); + + if (list_empty(&node->private_list)) + INIT_LIST_HEAD(&new_node->private_list); + else + list_replace(&node->private_list, &new_node->private_list); + + for (i = 0; i < XA_CHUNK_SIZE; i++) { + void *x = xa_entry_locked(xa, new_node, i); + + if (xa_is_node(x)) + rcu_assign_pointer(xa_to_node(x)->parent, new_node); + } + if (!new_node->parent) + slot = &xa->xa_head; + else + slot = &xa_parent_locked(xa, new_node)->slots[new_node->offset]; + rcu_assign_pointer(*slot, xa_mk_node(new_node)); + +unlock: + xa_unlock_irq(xa); + xa_node_free(node); + rcu_barrier(); +} + +static void xa_migrate(struct kmem_cache *s, void **objects, int nr, + int node, void *_unused) +{ + int i; + + for (i = 0; i < nr; i++) + xa_object_migrate(objects[i], node); +} + + void __init xarray_slabcache_init(void) { xa_node_cachep = kmem_cache_create("xarray_node", sizeof(struct xa_node), 0, SLAB_PANIC | SLAB_RECLAIM_ACCOUNT, xa_node_ctor); + kmem_cache_setup_mobility(xa_node_cachep, NULL, xa_migrate); } #ifdef XA_DEBUG From patchwork Mon May 20 05:40:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949927 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BA5B416C1 for ; Mon, 20 May 2019 05:42:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A6FC5285ED for ; Mon, 20 May 2019 05:42:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9B3652864A; Mon, 20 May 2019 05:42:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B532928646 for ; Mon, 20 May 2019 05:42:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 81AEB6B026A; Mon, 20 May 2019 01:42:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7CC126B026B; Mon, 20 May 2019 01:42:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 66C416B026C; Mon, 20 May 2019 01:42:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 491FE6B026A for ; Mon, 20 May 2019 01:42:32 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id w6so11802909qki.5 for ; Sun, 19 May 2019 22:42:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=HD+sjwMUXOsqB4lfcGjhKGGmBMq8bu2bkOc7CnZ0HxV+7V3EbpADtKP+5zeVS9/PB8 7pOpXnS7Iymec1qM6X4OA/jGI7L+RnVqUQbKKhkkFggxOIYbXnMJfQo5zulOOILEqSz/ 5jyxtFg74UQsebB4T65/WTaRmM32P0iD5XWgiCDpi6SGTXA3SVjiP9NcN2D1dnCoMP1N ZUza6DBMfCvpIQ1r7leQ1qq73ujQgxqOrj97Z5D5BfK8aLDVxsq061AgtBX7MesshwGB +BHFsQ0N5ADxFqCcg+ud8aV3azuZMxaG1xz6Af85bFOJVrpPQFMZ4T8Ru4yJnJxw8wvc tixg== X-Gm-Message-State: APjAAAXJb5WHlzITUX/rKPHVbIAPlkNHoTP5lkiIbOq0VHAUV+lO4Rkq 2BdonBh0pMtzgb918+eKI2AxxSO1Fg0Nv987P3aXvZEHBaxLNmbKVDiupZ3TAP5kLnqC0JPDskZ H96zRzAfcoPOqMalqLyutc2UFKRgC5qhd6WwEo+7inLPn7IXwPktktuFfa4DRgA0= X-Received: by 2002:a05:620a:15ac:: with SMTP id f12mr46486396qkk.311.1558330951977; Sun, 19 May 2019 22:42:31 -0700 (PDT) X-Google-Smtp-Source: APXvYqxEXBG+TWgHZ1EkUcp06hRcIpSB9ZPepzeOJTR8cjrkrLhnjkXTWk0lLWu8DbAuIA4AtUrp X-Received: by 2002:a05:620a:15ac:: with SMTP id f12mr46486336qkk.311.1558330950508; Sun, 19 May 2019 22:42:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330950; cv=none; d=google.com; s=arc-20160816; b=ias7/N0V7835fOkLi38d8m0jozG1++fI5Nababsf7VOwjqKFJeDVQAwDnjGS+H83vx FWNK3dTLDa9KbICJqRyh7dlZfeQjUswwQ/Cvj/q9D4Z5XTv9Vzp91jlCzicinCrPuIsV O45NI0lq/pZ0qdKUTTiTH5Hdxyr3xFblSdNAiLEd749IJpZsTQmd2hJDnkUQz1kEv36h ICgW5DgPZmLJWROH9GySIu9jeq8RSNQzrtXFIsv3PdHGjpuRBugX4NuwZENOLxjufKF5 hNcO5XuQCXb+jPQ5Fxa0SCUmaDcUnrf5zbQnvkOQ1O38RPRw5py3Wz3luwVbWg049shL VHPQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=Kh/xkqi280Y0pOqplTFUstR8Jesc5ScJN0sr32zOIZxUkTJfl/5nhVPm3X6ynS9xny H0mDdlWjtlC3Dd6z94fe7symFGMD9H/JFdZ/SK1nW5UXV8VRkeiP7xcgA5Wbx7tHZE8W ogK66LvGrOjoZEH3II9q6mpsmkkcHlIP7kAs2mTacu6psyqklru5YlK1f8LT7/R+OEkZ WcOVlq9mnNoQ//0tex39e+KE9Gyhq4ef5lype0LLSZfLqVMvAfcMVCo4/ls/UqDNAx0X p2OJdvcZVBX+hS1CyYyC1wS+4oFmW0G0ulEc5RNSvy10uOq65hY0oTc9yAIsKnja4jIl BDiQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=FP1+azQc; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id l58si10420483qvc.215.2019.05.19.22.42.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:42:30 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=FP1+azQc; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 3DB96D210; Mon, 20 May 2019 01:42:30 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:42:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=FP1+azQc rQo9GjVhupZBKSw2zglCLDgHlLDuQoAd2mhHz9NO4RMZnIck3CtwAzP/p0ddUgtn VuJd5rvuTRdb8D8ThKHXrNgTgMt0nG032738vlqt4bUylqfMz6CkhRHYHXIfgwSn 6NYAtjYgYyh0940MzBe8khndc1V4L8T3xeYYdt/gRW8Nb30ZlmwDTZT/JnLdDZI0 Ij+teHPg39csT2Osu1f114zoyoguVS18FbF+3qdM7IuPI3GFc3s6s7+uov0gHyCs ePW1AzP7C1mZyxIycSvVCr/fH4oC0ieZ9WULZkNjbWWKMqqJa6JrWCmn011VVuVX caINSxyKVfQeXA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepuddt X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 22A3580060; Mon, 20 May 2019 01:42:22 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 11/16] tools/testing/slab: Add XArray movable objects tests Date: Mon, 20 May 2019 15:40:12 +1000 Message-Id: <20190520054017.32299-12-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just implemented movable objects for the XArray. Let's test it intree. Add test module for the XArray's movable objects implementation. Functionality of the XArray Slab Movable Object implementation can usually be seen by simply by using `slabinfo` on a running machine since the radix tree is typically in use on a running machine and will have partial slabs. For repeated testing we can use the test module to run to simulate a workload on the XArray then use `slabinfo` to test object migration is functioning. If testing on freshly spun up VM (low radix tree workload) it may be necessary to load/unload the module a number of times to create partial slabs. Example test session -------------------- Relevant /proc/slabinfo column headers: name Prior to testing slabinfo report for radix_tree_node: # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8352 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 497 Sanity Checks : On Total: 8142848 SlabObj: 912 Full : 473 Redzoning : On Used : 4810752 SlabSiz: 16384 Partial: 24 Poisoning : On Loss : 3332096 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2806272 Align : 8 Objects: 17 Tracing : Off Lpadd: 437360 Here you can see the kernel was built with Slab Movable Objects enabled for the XArray (XArray uses the radix tree below the surface). After inserting the test module (note we have triggered allocation of a number of radix tree nodes increasing the object count but decreasing the number of partial slabs): # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8442 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 499 Sanity Checks : On Total: 8175616 SlabObj: 912 Full : 484 Redzoning : On Used : 4862592 SlabSiz: 16384 Partial: 15 Poisoning : On Loss : 3313024 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2836512 Align : 8 Objects: 17 Tracing : Off Lpadd: 439120 Now we can shrink the radix_tree_node cache: # slabinfo radix_tree_node --shrink # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8515 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 501 Sanity Checks : On Total: 8208384 SlabObj: 912 Full : 500 Redzoning : On Used : 4904640 SlabSiz: 16384 Partial: 1 Poisoning : On Loss : 3303744 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2861040 Align : 8 Objects: 17 Tracing : Off Lpadd: 440880 Note the single remaining partial slab. Signed-off-by: Tobin C. Harding --- tools/testing/slab/Makefile | 2 +- tools/testing/slab/slub_defrag_xarray.c | 211 ++++++++++++++++++++++++ 2 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 tools/testing/slab/slub_defrag_xarray.c diff --git a/tools/testing/slab/Makefile b/tools/testing/slab/Makefile index 440c2e3e356f..44c18d9a4d52 100644 --- a/tools/testing/slab/Makefile +++ b/tools/testing/slab/Makefile @@ -1,4 +1,4 @@ -obj-m += slub_defrag.o +obj-m += slub_defrag.o slub_defrag_xarray.o KTREE=../../.. diff --git a/tools/testing/slab/slub_defrag_xarray.c b/tools/testing/slab/slub_defrag_xarray.c new file mode 100644 index 000000000000..41143f73256c --- /dev/null +++ b/tools/testing/slab/slub_defrag_xarray.c @@ -0,0 +1,211 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SMOX_CACHE_NAME "smox_test" +static struct kmem_cache *cachep; + +/* + * Declare XArrays globally so we can clean them up on module unload. + */ + +/* Used by test_smo_xarray()*/ +DEFINE_XARRAY(things); + +/* Thing to store pointers to in the XArray */ +struct smox_thing { + long id; +}; + +/* It's up to the caller to ensure id is unique */ +static struct smox_thing *alloc_thing(int id) +{ + struct smox_thing *thing; + + thing = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!thing) + return ERR_PTR(-ENOMEM); + + thing->id = id; + return thing; +} + +/** + * smox_object_ctor() - SMO object constructor function. + * @ptr: Pointer to memory where the object should be constructed. + */ +void smox_object_ctor(void *ptr) +{ + struct smox_thing *thing = ptr; + + thing->id = -1; +} + +/** + * smox_cache_migrate() - kmem_cache migrate function. + * @cp: kmem_cache pointer. + * @objs: Array of pointers to objects to migrate. + * @size: Number of objects in @objs. + * @node: NUMA node where the object should be allocated. + * @private: Pointer returned by kmem_cache_isolate_func(). + */ +void smox_cache_migrate(struct kmem_cache *cp, void **objs, int size, + int node, void *private) +{ + struct smox_thing **ptrs = (struct smox_thing **)objs; + struct smox_thing *old, *new; + struct smox_thing *thing; + unsigned long index; + void *entry; + int i; + + for (i = 0; i < size; i++) { + old = ptrs[i]; + + new = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!new) { + pr_debug("kmem_cache_alloc failed\n"); + return; + } + + new->id = old->id; + + /* Update reference the brain dead way */ + xa_for_each(&things, index, thing) { + if (thing == old) { + entry = xa_store(&things, index, new, GFP_KERNEL); + if (entry != old) { + pr_err("failed to exchange new/old\n"); + return; + } + } + } + kmem_cache_free(cachep, old); + } +} + +/* + * test_smo_xarray() - Run some tests using an XArray. + */ +static int test_smo_xarray(void) +{ + const int keep = 6; /* Free 5 out of 6 items */ + const int nr_items = 10000; + struct smox_thing *thing; + unsigned long index; + void *entry; + int expected; + int i; + + /* + * Populate XArray, this adds to the radix_tree_node cache as + * well as the smox_test cache. + */ + for (i = 0; i < nr_items; i++) { + thing = alloc_thing(i); + entry = xa_store(&things, i, thing, GFP_KERNEL); + if (xa_is_err(entry)) { + pr_err("smox: failed to allocate entry: %d\n", i); + return -ENOMEM; + } + } + + /* Now free items, putting holes in both caches. */ + for (i = 0; i < nr_items; i++) { + if (i % keep == 0) + continue; + + thing = xa_erase(&things, i); + if (xa_is_err(thing)) + pr_err("smox: error erasing entry: %d\n", i); + kmem_cache_free(cachep, thing); + } + + expected = 0; + xa_for_each(&things, index, thing) { + if (thing->id != expected || index != expected) { + pr_err("smox: error; got %ld want %d at %ld\n", + thing->id, expected, index); + return -1; + } + expected += keep; + } + + /* + * Leave caches sparsely allocated. Shrink caches manually with: + * + * slabinfo radix_tree_node --shrink + * slabinfo smox_test --shrink + */ + + return 0; +} + +static int __init smox_cache_init(void) +{ + cachep = kmem_cache_create(SMOX_CACHE_NAME, + sizeof(struct smox_thing), + 0, 0, smox_object_ctor); + if (!cachep) + return -1; + + return 0; +} + +static void __exit smox_cache_cleanup(void) +{ + struct smox_thing *thing; + unsigned long i; + + xa_for_each(&things, i, thing) { + kmem_cache_free(cachep, thing); + } + xa_destroy(&things); + kmem_cache_destroy(cachep); +} + +static int __init smox_init(void) +{ + int ret; + + ret = smox_cache_init(); + if (ret) { + pr_err("smo_xarray: failed to create cache\n"); + return ret; + } + pr_info("smo_xarray: created kmem_cache: %s\n", SMOX_CACHE_NAME); + + kmem_cache_setup_mobility(cachep, NULL, smox_cache_migrate); + pr_info("smo_xarray: kmem_cache %s defrag enabled\n", SMOX_CACHE_NAME); + + /* + * Running this test consumes memory unless you shrink the + * radix_tree_node cache manually with `slabinfo`. + */ + ret = test_smo_xarray(); + if (ret) + pr_warn("test_smo_xarray failed: %d\n", ret); + + pr_info("smo_xarray: module loaded successfully\n"); + return 0; +} +module_init(smox_init); + +static void __exit smox_exit(void) +{ + smox_cache_cleanup(); + + pr_info("smo_xarray: module removed\n"); +} +module_exit(smox_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Tobin C. Harding"); +MODULE_DESCRIPTION("SMO XArray test module."); From patchwork Mon May 20 05:40:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949931 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DFF2116C1 for ; Mon, 20 May 2019 05:42:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF0AA28646 for ; Mon, 20 May 2019 05:42:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C23022864B; Mon, 20 May 2019 05:42:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DCCC728646 for ; Mon, 20 May 2019 05:42:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9EF386B026C; Mon, 20 May 2019 01:42:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9A0426B026D; Mon, 20 May 2019 01:42:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 840D36B026E; Mon, 20 May 2019 01:42:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 648F56B026C for ; Mon, 20 May 2019 01:42:39 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id y13so13223217qtc.7 for ; Sun, 19 May 2019 22:42:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=9XDuSxwKrHTa5OuRmvXZIbQcLUkhgPbDvSR6tXiHu4g=; b=l8ayigKylHO8o5S3u0WCHrxk8lMiHKmRh8ZvlRVcRTksiyxfDliECUKdO/ZFyegYnI 85hGsUhL7dBdhJ7yRV/j1NVPPBROC1t+EUHVpH76CjTYEW9vubaCXzrPqFfmAzTom1pM aTdgPCH/qqnZ6vNWL/ge/NjnmvNxY9vbnqB+Wr4tvv89LqjUSGO0eznYlDxcMZeXh4vS 2vbpplvWzfU16h0fmA2LAF0wyyAqEQYgWYu2g8r6QQw7bjijVM+WWjKLnT/1G08fvKNL ASAxjCPc/ie3eBBmSJbAQIcmnvc9XiWoMny+qk2t9cWBCTbNRDsPtBULlUNAt3q0Unx6 G41g== X-Gm-Message-State: APjAAAXFkzOnkLq1sHSwJzQrIDQWS3jzcJpav45KRXTXh/Iva2QPNAfK MC5cLPdvtckdRWPUpHGJGG+rGQz18hajsaZfRQqCb6+QXnMhE3A+eDAmtTfaquQJiPHcLAG7a2H 9PGyEN8G6iiaObPCtahYEooudrcIpWBngyXMe85NWo6w5PZrKVF/Oub5SHbLz+4Q= X-Received: by 2002:a37:dcc1:: with SMTP id v184mr47418741qki.338.1558330959121; Sun, 19 May 2019 22:42:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqyA0M5Ks5QAEgArrzqYon8ercJEbpE2xYbRc4HBrjR1sc68TGHogphI9VBdF1s/loUqmSKs X-Received: by 2002:a37:dcc1:: with SMTP id v184mr47418681qki.338.1558330957768; Sun, 19 May 2019 22:42:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330957; cv=none; d=google.com; s=arc-20160816; b=nKjC+WgGTutG0Bc1NVmbYGAaWLrOnM0j60diM62xbcX+s/4xNJXuDZadbuMqd2laCB DqrzAEyM0wf8yKRDEdRtH12IigoAiwjyB94e13FLtF/vVkj/g8Gfmk7CWvTS3RfryyXX AveUi+iEkvVjgIMdneOH3DC3CN3qFBFOMDUf3NVCaTlSDE2yJW3PsnyK06aUSUe4h4da UDA19mX2gNxxOOP2qUzOMU6/0Gnhp+BOY/JyL5Hu0KllmZHJW/o6/3Ok4RnwE95iUDff JZggKLt41iaJ5SY2mF20OnbkoJFUd/wmSYZP3zJQRqrqphZjZXqVkdKWavRfhz2NTFP1 Qk5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=9XDuSxwKrHTa5OuRmvXZIbQcLUkhgPbDvSR6tXiHu4g=; b=R0BEXY39MG60/pZIMKt1A/WT7gxKnExy2zQbCubf1OTwr8ySofHvLmFlS7qw+w6i+c D+4VOqPbLLRQgn9x/DRoEJSjNOJM0mTPraZ7GI/HqVQONmJUs4HnzTvtgZvSMtclINVn XePWXQ+X1yalVz7GFtho9675879BpbduMMihiy39JizL+9T7Q6dXYQmdZ4TmH6qU866+ Yz91y9e4WKtT2c4mgsx7xvsrHRDHl0W6oYNFFzfzwuSJkATS+yCSDkWNYLFS7dQqr5zq IfjP7Ge0Q9DPTuCzJPzEmw15GtCM5Zs7UGh9FjE72qT/+mrtzWz7rUdhPNguudApSOR/ JpEw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ZcsCbH2n; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id v63si1558026qkc.268.2019.05.19.22.42.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:42:37 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ZcsCbH2n; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 7AE6FDC02; Mon, 20 May 2019 01:42:37 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:42:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=9XDuSxwKrHTa5OuRmvXZIbQcLUkhgPbDvSR6tXiHu4g=; b=ZcsCbH2n kx1kNHGoZK7ikUL9wnmYUEIsxrCWj1knqAFlyLdJaK9uiLS4iwz5G6z8XnewR98P De3WXWOu9CSDdOa9H4CPAzqGZhflIdG8VTY1+mkbcNeVpIoMLEqoaiCKuEDKQOHq HitUxfue0bnJaGOnMPDRx7f5SXaX4p/RBYpc1kJKQNcn9BqhxRNp5wyMNC1K8Fp8 M4o6TSgZ+aaec/tmYMbuJ2fdiNGwLkapRM7eW3288aSZR+rc2lr0UlsRxsHRWE4u G+tcdDg2vlwDKOCGt2/jhDtYvydr8vwlB8kSArZcUjXvmFFU4spRPENzaXHDQsAT nq5LQTuebmFuXA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepuddu X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 5638E8005B; Mon, 20 May 2019 01:42:30 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 12/16] slub: Enable moving objects to/from specific nodes Date: Mon, 20 May 2019 15:40:13 +1000 Message-Id: <20190520054017.32299-13-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We have just implemented Slab Movable Objects (object migration). Currently object migration is used to defrag a cache. On NUMA systems it would be nice to be able to control the source and destination nodes when moving objects. Add CONFIG_SMO_NODE to guard this feature. CONFIG_SMO_NODE depends on CONFIG_SLUB_DEBUG because we use the full list. Implement moving all objects (including those in full slabs) to a specific node. Expose this functionality to userspace via a sysfs entry. Add sysfs entry: /sysfs/kernel/slab//move With this users get access to the following functionality: - Move all objects to specified node. echo "N1" > move - Move all objects from specified node to other specified node (from N1 -> to N2): echo "N1 N2" > move This also enables shrinking slabs on a specific node: echo "N1 N1" > move Signed-off-by: Tobin C. Harding --- mm/Kconfig | 7 ++ mm/slub.c | 249 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 256 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index ee8d1f311858..aa8d60e69a01 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -258,6 +258,13 @@ config ARCH_ENABLE_THP_MIGRATION config CONTIG_ALLOC def_bool (MEMORY_ISOLATION && COMPACTION) || CMA +config SMO_NODE + bool "Enable per node control of Slab Movable Objects" + depends on SLUB && SYSFS + select SLUB_DEBUG + help + On NUMA systems enable moving objects to and from a specified node. + config PHYS_ADDR_T_64BIT def_bool 64BIT diff --git a/mm/slub.c b/mm/slub.c index 2157205df7ba..9582f2fc97d2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4336,6 +4336,106 @@ static void move_slab_page(struct page *page, void *scratch, int node) s->migrate(s, vector, count, node, private); } +#ifdef CONFIG_SMO_NODE +/* + * kmem_cache_move() - Attempt to move all slab objects. + * @s: The cache we are working on. + * @node: The node to move objects away from. + * @target_node: The node to move objects on to. + * + * Attempts to move all objects (partial slabs and full slabs) to target + * node. + * + * Context: Takes the list_lock. + * Return: The number of slabs remaining on node. + */ +static unsigned long kmem_cache_move(struct kmem_cache *s, + int node, int target_node) +{ + struct kmem_cache_node *n = get_node(s, node); + LIST_HEAD(move_list); + struct page *page, *page2; + unsigned long flags; + void **scratch; + + if (!s->migrate) { + pr_warn("%s SMO not enabled, cannot move objects\n", s->name); + goto out; + } + + scratch = alloc_scratch(s); + if (!scratch) + goto out; + + spin_lock_irqsave(&n->list_lock, flags); + + list_for_each_entry_safe(page, page2, &n->partial, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + if (page->inuse) { + list_move(&page->lru, &move_list); + /* Stop page being considered for allocations */ + n->nr_partial--; + page->frozen = 1; + + slab_unlock(page); + } else { /* Empty slab page */ + list_del(&page->lru); + n->nr_partial--; + slab_unlock(page); + discard_slab(s, page); + } + } + list_for_each_entry_safe(page, page2, &n->full, lru) { + if (!slab_trylock(page)) + continue; + + list_move(&page->lru, &move_list); + page->frozen = 1; + slab_unlock(page); + } + + spin_unlock_irqrestore(&n->list_lock, flags); + + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + + /* Bail here to save taking the list_lock */ + if (list_empty(&move_list)) + goto out; + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + if (page->inuse == page->objects) { + list_add(&page->lru, &n->full); + slab_unlock(page); + } else { + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); +out: + return atomic_long_read(&n->nr_slabs); +} +#endif /* CONFIG_SMO_NODE */ + /* * kmem_cache_defrag() - Defragment node. * @s: cache we are working on. @@ -4450,6 +4550,32 @@ static unsigned long kmem_cache_defrag(struct kmem_cache *s, return n->nr_partial; } +#ifdef CONFIG_SMO_NODE +/* + * kmem_cache_move_to_node() - Move all slab objects to node. + * @s: The cache we are working on. + * @node: The target node to move objects to. + * + * Attempt to move all slab objects from all nodes to @node. + * + * Return: The total number of slabs left on emptied nodes. + */ +static unsigned long kmem_cache_move_to_node(struct kmem_cache *s, int node) +{ + unsigned long left = 0; + int nid; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == node) + continue; + + left += kmem_cache_move(s, nid, node); + } + + return left; +} +#endif + /** * kmem_defrag_slabs() - Defrag slab caches. * @node: The node to defrag or -1 for all nodes. @@ -5594,6 +5720,126 @@ static ssize_t shrink_store(struct kmem_cache *s, } SLAB_ATTR(shrink); +#ifdef CONFIG_SMO_NODE +static ssize_t move_show(struct kmem_cache *s, char *buf) +{ + return 0; +} + +/* + * parse_move_store_input() - Parse buf getting integer arguments. + * @buf: Buffer to parse. + * @length: Length of @buf. + * @arg0: Return parameter, first argument. + * @arg1: Return parameter, second argument. + * + * Parses the input from user write to sysfs file 'move'. Input string + * should contain either one or two node specifiers of form Nx where x + * is an integer specifying the NUMA node ID. 'N' or 'n' may be used. + * n/N may be omitted. + * + * e.g. + * echo 'N1' > /sysfs/kernel/slab/cache/move + * or + * echo 'N0 N2' > /sysfs/kernel/slab/cache/move + * + * Regex matching accepted forms: '[nN]?[0-9]( [nN]?[0-9])?' + * + * FIXME: This is really fragile. Input must be exactly correct, + * spurious whitespace causes parse errors. + * + * Return: 0 if an argument was successfully converted, or an error code. + */ +static ssize_t parse_move_store_input(const char *buf, size_t length, + long *arg0, long *arg1) +{ + char *s, *save, *ptr; + int ret = 0; + + if (!buf) + return -EINVAL; + + s = kstrdup(buf, GFP_KERNEL); + if (!s) + return -ENOMEM; + save = s; + + if (s[length - 1] == '\n') { + s[length - 1] = '\0'; + length--; + } + + ptr = strsep(&s, " "); + if (!ptr || strcmp(ptr, "") == 0) { + ret = 0; + goto out; + } + + if (*ptr == 'N' || *ptr == 'n') + ptr++; + ret = kstrtol(ptr, 10, arg0); + if (ret < 0) + goto out; + + if (s) { + if (*s == 'N' || *s == 'n') + s++; + ret = kstrtol(s, 10, arg1); + if (ret < 0) + goto out; + } + + ret = 0; +out: + kfree(save); + return ret; +} + +static bool is_valid_node(int node) +{ + int nid; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == node) + return true; + } + return false; +} + +/* + * move_store() - Move objects between nodes. + * @s: The cache we are working on. + * @buf: String received. + * @length: Length of @buf. + * + * Writes to /sys/kernel/slab//move are interpreted as follows: + * + * echo "N1" > move : Move all objects (from all nodes) to node 1. + * echo "N0 N1" > move : Move all objects from node 0 to node 1. + * + * 'N' may be omitted: + */ +static ssize_t move_store(struct kmem_cache *s, const char *buf, size_t length) +{ + long arg0 = -1; + long arg1 = -1; + int ret; + + ret = parse_move_store_input(buf, length, &arg0, &arg1); + if (ret < 0) + return -EINVAL; + + if (is_valid_node(arg0) && is_valid_node(arg1)) + (void)kmem_cache_move(s, arg0, arg1); + else if (is_valid_node(arg0)) + (void)kmem_cache_move_to_node(s, arg0); + + /* FIXME: What should we be returning here? */ + return length; +} +SLAB_ATTR(move); +#endif /* CONFIG_SMO_NODE */ + #ifdef CONFIG_NUMA static ssize_t remote_node_defrag_ratio_show(struct kmem_cache *s, char *buf) { @@ -5718,6 +5964,9 @@ static struct attribute *slab_attrs[] = { &reclaim_account_attr.attr, &destroy_by_rcu_attr.attr, &shrink_attr.attr, +#ifdef CONFIG_SMO_NODE + &move_attr.attr, +#endif &slabs_cpu_partial_attr.attr, #ifdef CONFIG_SLUB_DEBUG &total_objects_attr.attr, From patchwork Mon May 20 05:40:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949935 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B18081390 for ; Mon, 20 May 2019 05:42:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A01D628630 for ; Mon, 20 May 2019 05:42:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 945DD28649; Mon, 20 May 2019 05:42:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E049528630 for ; Mon, 20 May 2019 05:42:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8B6A6B026E; Mon, 20 May 2019 01:42:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A3D516B026F; Mon, 20 May 2019 01:42:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 92BC86B0270; Mon, 20 May 2019 01:42:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 71BFD6B026E for ; Mon, 20 May 2019 01:42:46 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id h4so13270977qtq.3 for ; Sun, 19 May 2019 22:42:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=+MHoJQqVfWBEgl6V34UkKqnW9RMSUkovsQXORa/Ihwg=; b=aUsm0564aiwA2hFdMZPGpDOeiMalp15Ir9Qq/BqoPbuOpki4oTQl9rlQ12rQ5tp1cK yvc8OOy1raBVVwWwH95ABOMGC2guYR8ffjQezR9CB3krsoRW0Tuks2f3+kBRoFKkYTFy +BSat0bbhBZWOyCP6W2wnhlP8wRlep8NKUwYVVekhCe8iPfEALcCKlyGcxtJn0HWm0rW ZLz7D717gyHQvMf5Z0Y+qq0iupchmFVA7ELJY5c23aW603dlJniHHhvH0dzKmlHqpMp4 31zfdc5XlCgoWALBq8vEJ3hbuxqXGeqwTPbXTUGbKnsNXKxHtO5WYNA+iJmb/lKIVqUg NQSA== X-Gm-Message-State: APjAAAU9nP8hkJ9miyJM0V/n7nvmgjlewNGewCqPnSjSWI88GgSxTFH3 oyTlMxhqo7wHOidBnWI1KSrlhVvpRn0T6O6v5IHANei5wc3nJLcCONyVKco0ZF7bVMKJKHdqW7l mVxriQsXQeU9XVZqMAMVn0AKkhjOiqgyhY6OksF5O9eds9MASjlfr6/bA5UnbE0Q= X-Received: by 2002:ac8:5218:: with SMTP id r24mr21864398qtn.177.1558330966191; Sun, 19 May 2019 22:42:46 -0700 (PDT) X-Google-Smtp-Source: APXvYqzB980/laFvBlyri34W1W/IuNJBUJB5JJ2+err6cmtn1XEMdVBzFUX7aidP9/BfpwT5QxRu X-Received: by 2002:ac8:5218:: with SMTP id r24mr21864344qtn.177.1558330964919; Sun, 19 May 2019 22:42:44 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330964; cv=none; d=google.com; s=arc-20160816; b=lyLSrM2CwxcQp9mYmDSYPz7L8c/0YulHehKDVVkCez0T092Vt4NH89xcrr923ZrTZX uLEx9D7mZ+4uD0SeAMUaTyiAuSRcquQBSL6k8+ssrM+tk3dFH6HPwmJlORzyz70cmNOn desLIEk8/1jMIfejqgSwXpA9OsRUO7pv7z2z4kfNIC4vyxsc3dcHTCWVT4vaQgLK+y/V 2G0o27W4TtSvIRdCm6lsmxc1TnD3yHBZUgE1/VDjX9A1T2BzMHawcLY2XwbNDlmekWkc VtLZCNXVha9Wb8IAJagGYDxCdTfJdTOx/JF1dzJamxO123YR0g4y0LHYVkhCVkBxhN75 DAIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=+MHoJQqVfWBEgl6V34UkKqnW9RMSUkovsQXORa/Ihwg=; b=YbhLfZxOvujy3s/PhUVhk7v/wEZAsgAB0kcpvyfaWL4IBwkPpmCEHeUwEx56VK0Zkw S/O3Prih4G48V2OjlKfjMAxi8ItHS0Q2G70P/j2kAwWbAGw162rM7yW8j89rvm1/c4jv LNZR9H/TQKPzXAirjQxn8m8tALLoReFvCB5W4dEe/3QKavJY92q2dwRNBAVkeNc/8DVL vzCKz6GIxZ6G4+myLVuYgbUlBdGPEaE+Ud3NkmeaBrK34khhqyweWWt66kv0k4bBxJZF pdsgdUukQcvcaFJOb17C5opLEQaSljOHsP4JLSMpfEZ2DqisLsinKaXNmcnufyoceRRB kEpw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=OH5SuBaX; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id p50si8818707qta.152.2019.05.19.22.42.44 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:42:44 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=OH5SuBaX; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 9DD9EF70A; Mon, 20 May 2019 01:42:44 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:42:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=+MHoJQqVfWBEgl6V34UkKqnW9RMSUkovsQXORa/Ihwg=; b=OH5SuBaX TPeWZdZ2LHrhURIZxeT4YNv8Mb3OH2xu451Ritc63buiCSMtLPLAf/flQMTXsodV fG/fYkdm4HiKKbE7cPuLKT5GhX6cyeOBw2TqFFch7AFEG1ADwFJMziSL3ot/LXRP lzEbzPCrgy4o6u8wNcFpuZDydWVAoKgc1EtgHcO6yds7N+6QhhAtj4KjfWdqSlIs hjpxa7KenHDmVQRfPXGPZfJJF2Qf26SO3vu1Vd2YopV6OKI+BSQKlpZ5xwmYwZn2 m3Z3EVTZJyoIWr/c9b5rU4ulpRbzoTf/UiHSbmJOwWVrsRTQf4uXlbZzFl2RsbtT Pde5ZnF1T9o4Jw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepuddv X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 8BB218005B; Mon, 20 May 2019 01:42:37 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 13/16] slub: Enable balancing slabs across nodes Date: Mon, 20 May 2019 15:40:14 +1000 Message-Id: <20190520054017.32299-14-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We have just implemented Slab Movable Objects (SMO). On NUMA systems slabs can become unbalanced i.e. many slabs on one node while other nodes have few slabs. Using SMO we can balance the slabs across all the nodes. The algorithm used is as follows: 1. Move all objects to node 0 (this has the effect of defragmenting the cache). 2. Calculate the desired number of slabs for each node (this is done using the approximation nr_slabs / nr_nodes). 3. Loop over the nodes moving the desired number of slabs from node 0 to the node. Feature is conditionally built in with CONFIG_SMO_NODE, this is because we need the full list (we enable SLUB_DEBUG to get this). Future version may separate final list out of SLUB_DEBUG. Expose this functionality to userspace via a sysfs entry. Add sysfs entry: /sysfs/kernel/slab//balance Write of '1' to this file triggers balance, no other value accepted. This feature relies on SMO being enable for the cache, this is done with a call to, after the isolate/migrate functions have been defined. kmem_cache_setup_mobility(s, isolate, migrate) Signed-off-by: Tobin C. Harding --- mm/slub.c | 120 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index 9582f2fc97d2..25b6d1e408e3 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4574,6 +4574,109 @@ static unsigned long kmem_cache_move_to_node(struct kmem_cache *s, int node) return left; } + +/* + * kmem_cache_move_slabs() - Attempt to move @num slabs to target_node, + * @s: The cache we are working on. + * @node: The node to move objects from. + * @target_node: The node to move objects to. + * @num: The number of slabs to move. + * + * Attempts to move @num slabs from @node to @target_node. This is done + * by migrating objects from slabs on the full_list. + * + * Return: The number of slabs moved or error code. + */ +static long kmem_cache_move_slabs(struct kmem_cache *s, + int node, int target_node, long num) +{ + struct kmem_cache_node *n = get_node(s, node); + LIST_HEAD(move_list); + struct page *page, *page2; + unsigned long flags; + void **scratch; + long done = 0; + + if (node == target_node) + return -EINVAL; + + scratch = alloc_scratch(s); + if (!scratch) + return -ENOMEM; + + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &n->full, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + list_move(&page->lru, &move_list); + page->frozen = 1; + slab_unlock(page); + + if (++done >= num) + break; + } + spin_unlock_irqrestore(&n->list_lock, flags); + + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + /* + * This is best effort only, if slab still has + * objects just put it back on the partial list. + */ + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); + + return done; +} + +/* + * kmem_cache_balance_nodes() - Balance slabs across nodes. + * @s: The cache we are working on. + */ +static void kmem_cache_balance_nodes(struct kmem_cache *s) +{ + struct kmem_cache_node *n = get_node(s, 0); + unsigned long desired_nr_slabs_per_node; + unsigned long nr_slabs; + int nr_nodes = 0; + int nid; + + (void)kmem_cache_move_to_node(s, 0); + + for_each_node_state(nid, N_NORMAL_MEMORY) + nr_nodes++; + + nr_slabs = atomic_long_read(&n->nr_slabs); + desired_nr_slabs_per_node = nr_slabs / nr_nodes; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == 0) + continue; + + kmem_cache_move_slabs(s, 0, nid, desired_nr_slabs_per_node); + } +} #endif /** @@ -5838,6 +5941,22 @@ static ssize_t move_store(struct kmem_cache *s, const char *buf, size_t length) return length; } SLAB_ATTR(move); + +static ssize_t balance_show(struct kmem_cache *s, char *buf) +{ + return 0; +} + +static ssize_t balance_store(struct kmem_cache *s, + const char *buf, size_t length) +{ + if (buf[0] == '1') + kmem_cache_balance_nodes(s); + else + return -EINVAL; + return length; +} +SLAB_ATTR(balance); #endif /* CONFIG_SMO_NODE */ #ifdef CONFIG_NUMA @@ -5966,6 +6085,7 @@ static struct attribute *slab_attrs[] = { &shrink_attr.attr, #ifdef CONFIG_SMO_NODE &move_attr.attr, + &balance_attr.attr, #endif &slabs_cpu_partial_attr.attr, #ifdef CONFIG_SLUB_DEBUG From patchwork Mon May 20 05:40:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949939 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 63A3F17D2 for ; Mon, 20 May 2019 05:42:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 53E7928649 for ; Mon, 20 May 2019 05:42:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 480042864A; Mon, 20 May 2019 05:42:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B318228630 for ; Mon, 20 May 2019 05:42:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 88ECC6B0270; Mon, 20 May 2019 01:42:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 83E996B0271; Mon, 20 May 2019 01:42:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 706D86B0272; Mon, 20 May 2019 01:42:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 4E56F6B0270 for ; Mon, 20 May 2019 01:42:53 -0400 (EDT) Received: by mail-qt1-f199.google.com with SMTP id 49so13156466qtn.23 for ; Sun, 19 May 2019 22:42:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=KJQThgKBrb4Q/HDch0M4apqEnwzOiy7Mz4o6TMTuCfU=; b=VSnTAfELAtZjT7DiABm2da+TXil5j6ky8JVxTYWehKvOeOC6P0521aCiMtbMqKuJdD Ezo48mmcaemmcd1f/NKRgXEbWhBzG87xTOAmx6Cm74b5KL684fX65n6JBalPav02nY3J N5428vcQQzUoNcsMIquCpElcyKG9sdWD9Zz61wbxGJcehF860EtXk0ksJOed8cklNEo1 6DQsesqTfP03HbDKS+Gq5daM8bZj7m9i6IzvA6W9h4O7E1SG5Vfn4jSgZu2oVlPvdYrX B+vThHC4LI7K9/UPPkjjhcDPdfvm35g8JA9cddsZIuCpd3ldIzyf1znX6Q5ahsNJSSt/ eAug== X-Gm-Message-State: APjAAAUcLUHpqIx/yXIv4q9UHqhiRcW+nTVaJPI3AEO0dgm32KfZYwUz kxlsVBvlUP2UJVf0NQtyKE7YDEm7sjOXQrcpiXDIzR8zOJAzGOXDdjXxDKq2Up33dmoOv20nBVF 0SqMEw3Q5DTzmj38+kSx55ZJudgvyElga6gtSveOJB5v26X5Nbi5gZRN0imz9238= X-Received: by 2002:aed:237b:: with SMTP id i56mr43935232qtc.370.1558330973087; Sun, 19 May 2019 22:42:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqysDVLSmCJLq9f0Hy5N0GN9GFho345ZqULVgwrtM+b3msoVq9GG8f4iJNZm/x3w6SK2+HxO X-Received: by 2002:aed:237b:: with SMTP id i56mr43935192qtc.370.1558330972168; Sun, 19 May 2019 22:42:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330972; cv=none; d=google.com; s=arc-20160816; b=AVYrbbMJcb7a+GIW2hhw6NNVs7bPjW639XvZ15A41SNPa7cGH+r+DapB4amuhDDCNS xan4+DtQDBnuFyQS+tkwK6BnoMGrUI703MZvRTnGIEIyZiGrbiU1umR6QCJxpLUq9MVE Ac0/0p1ifTTTnLjoqf+tUQGQjWJrVyk0Lp7XO/uTgxysQFcEIojibidicy+x4DmEX2MO lWrsMV6CoysAEt9rIJ3QE+P8ZYxfRAhXaqNcv/HYFEW3zQarsA4FOKnl9A4cMPZskv3V t9rKZSofMGaH0ZeCC/ns5u55EihPaj+jr/TuhppzFiOwEf8BpA6KSNMQHTj3RBf6kGc7 kVrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=KJQThgKBrb4Q/HDch0M4apqEnwzOiy7Mz4o6TMTuCfU=; b=kZDQI/SRcAWfb+k1c6/bRHDz94WG6cjKKC8mv25pzLq2oU0cdX7XhGmTwxgc/A2JIH ztVsb/x0zIU4vO+EStxVfo+eKp/Yu0AKlSHdP22KcPdXa8YK0TwssGH/FE0RkeF3W5Yb 7/oOPBb1PhspB+hFjKvT6Yb0v9Bw0nGDGTe+U7a4wcRYG5pMCYlNF9tjt9HOOS1RwztX G3vtnUu/toqDyAwm/HiGMACswOB8brHXSzvWfFFYgE68RVAg+geAAoUpQtftGSNrP7C3 Ig/o3JojRr+cTWN+Uv9fU8FbckcMCweZ7QNuEr/pGMJYtpttL47TyQ4U2P5HLdLFw5EG aveA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=tuExDE8O; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id t8si772318qti.213.2019.05.19.22.42.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:42:52 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=tuExDE8O; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id DDC63101E4; Mon, 20 May 2019 01:42:51 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:42:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=KJQThgKBrb4Q/HDch0M4apqEnwzOiy7Mz4o6TMTuCfU=; b=tuExDE8O RkcZizOzolt761iWo9hMoZ8sDm2tcPebJBTnsY0OaLLfnyYN8biO1EdE51FuM1gb 3d2kD7w8qkpPqccJPkgo9sGipp7cWUnj6W2qCKQGBlVDEbaw9uyo+s5PPWM0oepo t+HHcUTiGINV34TQ15eeY5TWwCZHyK8axgikJjiquVb41leTKv3fh/dxzOCYqj6o rXjo0BvFTNd85SqshQplNuCxEi9O/dx7aT6o5U9bdU3GZVjR14SlDDAo25xfhV3l 7jAd+Po++qBSx1L7tYMq8pEMLjvXqg6JTX3bV56OLYDMShyfbQTMtlfSBCdkX0RO 3XiVa+os9y8niA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepudef X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id C22CE8005B; Mon, 20 May 2019 01:42:44 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 14/16] dcache: Provide a dentry constructor Date: Mon, 20 May 2019 15:40:15 +1000 Message-Id: <20190520054017.32299-15-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In order to support object migration on the dentry cache we need to have a determined object state at all times. Without a constructor the object would have a random state after allocation. Provide a dentry constructor. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index 8136bda27a1f..b7318615979d 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -1602,6 +1602,16 @@ void d_invalidate(struct dentry *dentry) } EXPORT_SYMBOL(d_invalidate); +static void dcache_ctor(void *p) +{ + struct dentry *dentry = p; + + /* Mimic lockref_mark_dead() */ + dentry->d_lockref.count = -128; + + spin_lock_init(&dentry->d_lock); +} + /** * __d_alloc - allocate a dcache entry * @sb: filesystem it will belong to @@ -1657,7 +1667,6 @@ struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) dentry->d_lockref.count = 1; dentry->d_flags = 0; - spin_lock_init(&dentry->d_lock); seqcount_init(&dentry->d_seq); dentry->d_inode = NULL; dentry->d_parent = dentry; @@ -3095,14 +3104,17 @@ static void __init dcache_init_early(void) static void __init dcache_init(void) { - /* - * A constructor could be added for stable state like the lists, - * but it is probably not worth it because of the cache nature - * of the dcache. - */ - dentry_cache = KMEM_CACHE_USERCOPY(dentry, - SLAB_RECLAIM_ACCOUNT|SLAB_PANIC|SLAB_MEM_SPREAD|SLAB_ACCOUNT, - d_iname); + slab_flags_t flags = + SLAB_RECLAIM_ACCOUNT | SLAB_PANIC | SLAB_MEM_SPREAD | SLAB_ACCOUNT; + + dentry_cache = + kmem_cache_create_usercopy("dentry", + sizeof(struct dentry), + __alignof__(struct dentry), + flags, + offsetof(struct dentry, d_iname), + sizeof_field(struct dentry, d_iname), + dcache_ctor); /* Hash may have been set up in dcache_init_early */ if (!hashdist) From patchwork Mon May 20 05:40:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949945 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3A97E16C1 for ; Mon, 20 May 2019 05:43:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2B3682862A for ; Mon, 20 May 2019 05:43:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1F5902864E; Mon, 20 May 2019 05:43:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D1D92862A for ; Mon, 20 May 2019 05:43:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 448686B0272; Mon, 20 May 2019 01:43:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3F8F06B0273; Mon, 20 May 2019 01:43:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E9556B0274; Mon, 20 May 2019 01:43:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id 07B026B0272 for ; Mon, 20 May 2019 01:43:01 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id h11so11803407qkk.1 for ; Sun, 19 May 2019 22:43:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=saCxizw32Jq7dMTN5gwfHcfa+dy7EpZHGbTHKFMG+ho=; b=aLKpSCqqsloO/JFiMsaJlQ6/H9WdZrtp3jeDniQUVMI7wARFAU9yg+Y84uQuH8U1cV UiwNlSboGtyeG3+ELarmVJRzGW6mHSRmOz/BiYcn+yBwpAuCr1AP0VVEMuueTerfUni+ EXgwN51Es1NYSb7MzO+DeXJHSX1x9yUIpGTZLb80yScvt1CrekPoYM8AaZoaBBMv0hii wUOJVE4xPSAubveZArlLEROJAxK1EYAcWUv1wLhOmrtfF+bXRPYL+bUXAH7w9t3McAEz 596y5+VLPd9/NLKv7qh0EZsBllqGW1YNUJwHJu8xJXOzCXyfsNICNWKdKATS8fh1DuGr 3Efg== X-Gm-Message-State: APjAAAX8WYPELMa+2/iKM8ZDKHnA7ky/87195iKCdzs/SeGfT1JuXqH5 ZIsvDxVQ4Y+3CdXW7f+U36DcY7AnMcjY+jNTE6uQ7tfaPjmB9lN8MDdkshoLa9SPBrokVpN2pTe lF50AyPWR4yU1ebfR25a6t8WWkF0d6hZJx20vcJs7JsD12HJMqeDEUQMdV5hjVZI= X-Received: by 2002:a0c:b92f:: with SMTP id u47mr49121300qvf.94.1558330980727; Sun, 19 May 2019 22:43:00 -0700 (PDT) X-Google-Smtp-Source: APXvYqx9d9F7XDuiHOHgVcj+XLX1lCU6U6ZHAyj8SKzqb/1VsYuBWC98ka9GNgQegqUjh72fQ4g6 X-Received: by 2002:a0c:b92f:: with SMTP id u47mr49121227qvf.94.1558330979547; Sun, 19 May 2019 22:42:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330979; cv=none; d=google.com; s=arc-20160816; b=xuUyp4OLmswxnOw0o2EvbLSYbarWfGY2DcTTD1o59YbZqUF9Oyp5UBXPuTuUn8LylW EEa5ROsnrhZuVehm2cMvvOWqyPz2874vvcKH/pFr9WfTD5Mfcz8GIFnOy3oR7ciCQpr/ WZRj4rq1wZYGqdDXEDQI8h8OsFh2YOmZGPXRazGno9gyZHy0W0xMJib1XaLTqlU4Y+w3 NJ6epaKJ8chrWm5LYfNrxyS0Op77onsCU+AQtRdUzR/uDqnJz21dYS+/FJNuwwvVTLS0 rNfuU9VeqFMnqoK/DvqhCCwxE1GYZJfkK5Z5T8vMZ4cb3LTCdm/jzw/qJtDESzkxPCHD fSEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=saCxizw32Jq7dMTN5gwfHcfa+dy7EpZHGbTHKFMG+ho=; b=IvfR/zh8Tunc79XvF2lNg8kblqthMZpxTFAX4jCwT+33HqznEfrd5BA7FvSBPbmViB tJO4mIYM6gkJJWeX9t9JqfQP78uiW7oJqul/ttLVa6hNab09+VtO8vQw0ld6ojU804La GBRvB4K+MOH1g/Eu2YilVfARvXlKIgyDu08Vlkip5W0bDE2It6bG8FSfiF+XKr9iYDN/ w78d2J9gxiVfZVPQTIM5gfWeYntAH0wVaVBsx6OsfwluGD4LyO8e/T/RPk5eGLJfMQol IlJ2x+MddAlxG5Gxdj+27odi5hsEenIrfDqyssePiRAPZYE9vD9FBKInkWj8uyQ29p1x wsxA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=gAlSX8TO; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id k36si3040568qtd.217.2019.05.19.22.42.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:42:59 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=gAlSX8TO; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 3D25710E56; Mon, 20 May 2019 01:42:59 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:42:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=saCxizw32Jq7dMTN5gwfHcfa+dy7EpZHGbTHKFMG+ho=; b=gAlSX8TO fcuek0YYCjnPtnRqtTcFPf18A3CjnScnL+unrhGzFdsGVFTur1RufeyGl9kcHIaL uh+hknblFftXo3OYtovnHQI2ovqIw3J2VlkVvKvL2XMGveghgWqujLvvqSY+MEoe E4fMy9EQy2zlr1MI5fcpEWgb9YE1vP/9Uql5R27Y98ZbHlNDeqFpIlf9d7j1De8p CK6f9780Qk7kRhUIHEU9zvE4Ugss5ENJOjjRl8UG/N8YxE2xrnBzHk9BvYQANfJN RHwzqYkX7doCeg+0YeDniqXxNMRyE+u8ftFvTnoIrEeAIoDQguHkwlYO5BE0GgMT zwwTNjFHjeRviw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepudeg X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 016CE8005B; Mon, 20 May 2019 01:42:51 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 15/16] dcache: Implement partial shrink via Slab Movable Objects Date: Mon, 20 May 2019 15:40:16 +1000 Message-Id: <20190520054017.32299-16-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The dentry slab cache is susceptible to internal fragmentation. Now that we have Slab Movable Objects we can attempt to defragment the dcache. Dentry objects are inherently _not_ relocatable however under some conditions they can be free'd. This is the same as shrinking the dcache but instead of shrinking the whole cache we only attempt to free those objects that are located in partially full slab pages. There is no guarantee that this will reduce the memory usage of the system, it is a compromise between fragmented memory and total cache shrinkage with the hope that some memory pressure can be alleviated. This is implemented using the newly added Slab Movable Objects infrastructure. The dcache 'migration' function is intentionally _not_ called 'd_migrate' because we only free, we do not migrate. Call it 'd_partial_shrink' to make explicit that no reallocation is done. Implement isolate and 'migrate' functions for the dentry slab cache. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/fs/dcache.c b/fs/dcache.c index b7318615979d..0dfe580c2d42 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -31,6 +31,7 @@ #include #include #include +#include #include "internal.h" #include "mount.h" @@ -3071,6 +3072,79 @@ void d_tmpfile(struct dentry *dentry, struct inode *inode) } EXPORT_SYMBOL(d_tmpfile); +/* + * d_isolate() - Dentry isolation callback function. + * @s: The dentry cache. + * @v: Vector of pointers to the objects to isolate. + * @nr: Number of objects in @v. + * + * The slab allocator is holding off frees. We can safely examine + * the object without the danger of it vanishing from under us. + */ +static void *d_isolate(struct kmem_cache *s, void **v, int nr) +{ + struct list_head *dispose; + struct dentry *dentry; + int i; + + dispose = kmalloc(sizeof(*dispose), GFP_KERNEL); + if (!dispose) + return NULL; + + INIT_LIST_HEAD(dispose); + + for (i = 0; i < nr; i++) { + dentry = v[i]; + spin_lock(&dentry->d_lock); + + if (dentry->d_lockref.count > 0 || + dentry->d_flags & DCACHE_SHRINK_LIST) { + spin_unlock(&dentry->d_lock); + continue; + } + + if (dentry->d_flags & DCACHE_LRU_LIST) + d_lru_del(dentry); + + d_shrink_add(dentry, dispose); + spin_unlock(&dentry->d_lock); + } + + return dispose; +} + +/* + * d_partial_shrink() - Dentry migration callback function. + * @s: The dentry cache. + * @_unused: We do not access the vector. + * @__unused: No need for length of vector. + * @___unused: We do not do any allocation. + * @private: list_head pointer representing the shrink list. + * + * Dispose of the shrink list created during isolation function. + * + * Dentry objects can _not_ be relocated and shrinking the whole dcache + * can be expensive. This is an effort to free dentry objects that are + * stopping slab pages from being free'd without clearing the whole dcache. + * + * This callback is called from the SLUB allocator object migration + * infrastructure in attempt to free up slab pages by freeing dentry + * objects from partially full slabs. + */ +static void d_partial_shrink(struct kmem_cache *s, void **_unused, int __unused, + int ___unused, void *private) +{ + struct list_head *dispose = private; + + if (!private) /* kmalloc error during isolate. */ + return; + + if (!list_empty(dispose)) + shrink_dentry_list(dispose); + + kfree(private); +} + static __initdata unsigned long dhash_entries; static int __init set_dhash_entries(char *str) { @@ -3116,6 +3190,8 @@ static void __init dcache_init(void) sizeof_field(struct dentry, d_iname), dcache_ctor); + kmem_cache_setup_mobility(dentry_cache, d_isolate, d_partial_shrink); + /* Hash may have been set up in dcache_init_early */ if (!hashdist) return; From patchwork Mon May 20 05:40:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10949949 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3D8D51390 for ; Mon, 20 May 2019 05:43:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D8722864A for ; Mon, 20 May 2019 05:43:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 214C32864E; Mon, 20 May 2019 05:43:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9DD7B2862A for ; Mon, 20 May 2019 05:43:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F3556B0274; Mon, 20 May 2019 01:43:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7A3C56B0275; Mon, 20 May 2019 01:43:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B9756B0276; Mon, 20 May 2019 01:43:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 4D30D6B0274 for ; Mon, 20 May 2019 01:43:08 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id q32so13217442qtk.10 for ; Sun, 19 May 2019 22:43:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=BWgf9mQL60frF07ii2U3Yx/gnin3umDFroib+xOOshw=; b=t4IqVD2JOJWfgEmS6HWhyoJcq6E0ok2NNvR/ndyNSFQKLVgnooHnC4KbSXO6DPBn2s 7disMB4OziPmjRK0h6ljoHUtwOYlzJodXkOgKacRHYSR2kXUZwj8sxb9FRmGAMUYB70q TzrFnnNtoz+MHRhyc9Aw+ITk6t4NjhLgyGQ/FlHEd1AVSeW0irlq/XWl/Tf0x8Ixgcxc UyjiTJ0Xr0ecEEXnKTFqaGiojfG4J3JqBlpy4YiTSFOYyTWot8iQum1WibP3QXIxgcoF Jq68AvXFkzslFywWeWQ8y4B3ls8IhOaDcVUBpwFJ91xyDTGDd2rA0rVZN0sI9c1Fhvzq TfCg== X-Gm-Message-State: APjAAAUKBdl8Jm00ZNpZcIJhgLVg4jzwAMie3WZGj/XedeEsO4CGOd1+ vP7A9g1cSyHg4ZYWdOMz/0B/efAyxykSn1wuGL8maCe9ISv0msAzDFfyxSBU/VgYY0evitp2Dfg IMoK9sLcZ9G+otd0+4qGfy7vjh8Ag3FeCs94Ts+/eId4iRY03vxMOcYI10YkLEoU= X-Received: by 2002:a37:b4c6:: with SMTP id d189mr57411648qkf.173.1558330988002; Sun, 19 May 2019 22:43:08 -0700 (PDT) X-Google-Smtp-Source: APXvYqwis8az85ijfJiecv/3cPCPGNUqV20X2uAqfUskYqXoMeTNK+vV+OTr+blh5Qd4zA0uAtaH X-Received: by 2002:a37:b4c6:: with SMTP id d189mr57411605qkf.173.1558330987036; Sun, 19 May 2019 22:43:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1558330987; cv=none; d=google.com; s=arc-20160816; b=jOs4Nkn2wapHZ3Qxr6+vdOmIKu8XD+VKA17Nbb7pV/zM1PfZwjE5YII8UjRRII6x0h x/Tcg0UprCiVNuTGc/kh9OKPP2N/e04bwEPzlGQU+IqhUGevbovqRCWFSCz5ZhdGSxev u8RF9xtjeaBLJ3LAz2iN7Bk7HJryde+3PazebRwbMPpFtUzMTh3Vgwnf2ERAqnA6MLBH g4B/vODtnBvDzkLjm+m7pFcysGyY5E6HGZFxpM2+twg4HNgh+51ZdoR6XJIVt6ESvjbk W4biBw0o2FL3ajWeYXOW9r6f9U/hq30BK+p1s6lKbuRBTtizt1LM4uA5XE2yaMmXaOKK RL/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=BWgf9mQL60frF07ii2U3Yx/gnin3umDFroib+xOOshw=; b=A6IXwPrw9L5q8Y/Bx/zd8Q06TlA+AWQt9DQYa1rXNLcbRVyK0Az6FqSVVC1sxG5dN2 +7jSlCrWmzhtNpkDFdnWWkH7es9XZzD941kVslhrzDGte75wtdhMTDbfIivTSmHkw0+E HCqM4YMs3uQSS3LJedRDoeUmLstoWk1VGQzUJQ3NuePG9o8haK8KkRpkx0nvD4j1xh0j v5SGO8m2C+4IgdDVrE8SrCGqJyeoBHuzJuBY/IfniGoR6XV6eCf9e5s6ZTelXlbLZ22r JqP0ppamnrFHaEWDp3YHqwIb9y5KrfGMmeW8gV25OF5RKp8XF4dfRiOVrIdGzH8Jwarn nm5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=LhKqHj9w; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new4-smtp.messagingengine.com (new4-smtp.messagingengine.com. [66.111.4.230]) by mx.google.com with ESMTPS id u5si2283273qta.21.2019.05.19.22.43.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 19 May 2019 22:43:07 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) client-ip=66.111.4.230; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=LhKqHj9w; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.230 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id BB657115CB; Mon, 20 May 2019 01:43:06 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 20 May 2019 01:43:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=BWgf9mQL60frF07ii2U3Yx/gnin3umDFroib+xOOshw=; b=LhKqHj9w zvjIxq0G2O5k2GycyyiBsamzi2rk2+Lf+YY+F5xR7uqTFhkUs01LFET99pHIduQs zOM4rZdOSvJ/+z966a1bZjVfEqSxqIESB2sj2qYi/MyuhQjzu91wrVGuA9y2tEt/ krfRq77gRiJ8tckBxpUTJnqfd3NJOGBYPJN2U39hcDUUkSJSWZSjJ0xSt1+Ea1O1 +jt7qGo8NnY9WOhtrB/Q8GgsmaFNzTIQ9zSeGHSaxKOlKiD0s5oDmFpMAD46hK7W JEhgU6aa+NeSITEQ9r1zxvFsz532rqKkHToJnh6sHAdJRobsuaRB9jd+5RQ66NuX WpprC/lEu1nFAw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddtjedguddtudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgs ihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenuc fkphepuddvgedrudeiledrudehiedrvddtfeenucfrrghrrghmpehmrghilhhfrhhomhep thhosghinheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepudeh X-ME-Proxy: Received: from eros.localdomain (124-169-156-203.dyn.iinet.net.au [124.169.156.203]) by mail.messagingengine.com (Postfix) with ESMTPA id 3664D80060; Mon, 20 May 2019 01:42:58 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton , Matthew Wilcox Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , Theodore Ts'o , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v5 16/16] dcache: Add CONFIG_DCACHE_SMO Date: Mon, 20 May 2019 15:40:17 +1000 Message-Id: <20190520054017.32299-17-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190520054017.32299-1-tobin@kernel.org> References: <20190520054017.32299-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In an attempt to make the SMO patchset as non-invasive as possible add a config option CONFIG_DCACHE_SMO (under "Memory Management options") for enabling SMO for the DCACHE. Whithout this option dcache constructor is used but no other code is built in, with this option enabled slab mobility is enabled and the isolate/migrate functions are built in. Add CONFIG_DCACHE_SMO to guard the partial shrinking of the dcache via Slab Movable Objects infrastructure. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 4 ++++ mm/Kconfig | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/fs/dcache.c b/fs/dcache.c index 0dfe580c2d42..96063e872366 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -3072,6 +3072,7 @@ void d_tmpfile(struct dentry *dentry, struct inode *inode) } EXPORT_SYMBOL(d_tmpfile); +#ifdef CONFIG_DCACHE_SMO /* * d_isolate() - Dentry isolation callback function. * @s: The dentry cache. @@ -3144,6 +3145,7 @@ static void d_partial_shrink(struct kmem_cache *s, void **_unused, int __unused, kfree(private); } +#endif /* CONFIG_DCACHE_SMO */ static __initdata unsigned long dhash_entries; static int __init set_dhash_entries(char *str) @@ -3190,7 +3192,9 @@ static void __init dcache_init(void) sizeof_field(struct dentry, d_iname), dcache_ctor); +#ifdef CONFIG_DCACHE_SMO kmem_cache_setup_mobility(dentry_cache, d_isolate, d_partial_shrink); +#endif /* Hash may have been set up in dcache_init_early */ if (!hashdist) diff --git a/mm/Kconfig b/mm/Kconfig index aa8d60e69a01..7dcea76e5ecc 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -265,6 +265,13 @@ config SMO_NODE help On NUMA systems enable moving objects to and from a specified node. +config DCACHE_SMO + bool "Enable Slab Movable Objects for the dcache" + depends on SLUB + help + Under memory pressure we can try to free dentry slab cache objects from + the partial slab list if this is enabled. + config PHYS_ADDR_T_64BIT def_bool 64BIT