From patchwork Wed Apr 3 04:21:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882741 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 691D01575 for ; Wed, 3 Apr 2019 04:22:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4CC3F287A6 for ; Wed, 3 Apr 2019 04:22:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3C95A289BF; Wed, 3 Apr 2019 04:22:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 676D1287A6 for ; Wed, 3 Apr 2019 04:22:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 449D76B026F; Wed, 3 Apr 2019 00:22:53 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 421C16B0272; Wed, 3 Apr 2019 00:22:53 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3129C6B0274; Wed, 3 Apr 2019 00:22:53 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 127AB6B026F for ; Wed, 3 Apr 2019 00:22:53 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id p26so15548919qtq.21 for ; Tue, 02 Apr 2019 21:22:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ToWY0R45y8BHjmAdWxteY+oDuIkdik3lI8PSNGM4h2M=; b=YbPb9BG3D34xOX8lhpj1L5Ewgkjm6AIc0X1E5SrQJjo3J8yifOLIkRJH0PbWQXIE8+ zhdnbAjMamQ/ZRXmzWC1UabGNi8eZXc/gYFNcuOfavt53f3AwSgSwOb9J0Cfs6XCwC54 5CdpErm+jdEDj/YlQv/WhEhCManHanfmskEEOP8LTubjAghM2OTBcuiCVBA824H/Z+vI gldUrnVrg6HM7hFwEHWtU+YMD2u6ETMV2atVdGAWZJsdD4W022z//y2kK6PjR7L4DuDT 4Pon+bhawHTPufwhhav1KYw440tboIM2S3U3q4Ao4a8xW8p8MYkqac3glqVT++BKFUAa 8oxg== X-Gm-Message-State: APjAAAVoOKdtqngOCNzauNyEV1Vzq+mmkB7BjMDXWdqaDBKAB9DIWzjj T1nB2AY1aXw4fSDyKwDj4fL4mBtDxNyijdJWtX9aasXqvwiAs3Rk05fsWhf5LkiS1giEYzT6TiL xW+36onu1b4PvYhrtnb5bhEHcMhb7Qxtlz/YBFxk7Dy8gx/ddjNC53e2L25zkJYQ= X-Received: by 2002:a0c:b99c:: with SMTP id v28mr28089913qvf.10.1554265372800; Tue, 02 Apr 2019 21:22:52 -0700 (PDT) X-Google-Smtp-Source: APXvYqwl27u43friSBtI/4bQ28lq12bQ3dV4XhYvbDVdUM/qP7eNnD5GaqBnLmmYl0kUiepMdVXY X-Received: by 2002:a0c:b99c:: with SMTP id v28mr28089880qvf.10.1554265371696; Tue, 02 Apr 2019 21:22:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265371; cv=none; d=google.com; s=arc-20160816; b=Nj4HZgu/kJ7I873A6uGNbnmTGFMgoZcjCJuvcVfjKPIl1cHgyR6mP8V12uIn8OpyBR epB0hVhqa1qJlUo2fjq1rEK35pn2Em4BmgBKfXKlu8tJchrIBHQj+Cg2qjnkdFUG6AW+ 1WhwPT/4H0yjDn6UiIuhrzbicDD5Me1JRRKPzwS6J5VoBCdP3KgA+LQiN16HavOdwS5r bL6Xzoxh7gxBaR/p5ngOa5p32scz2oX//51bfIl28z/3ImvOP/lbT40gCKP5JXgF5oH9 lA0XN2dD7hWFDFdhhskeIuq+6V6iNSbeyrGWdZCfMrvYhY+QLWbcpdo7VwGp8n8ofcmy GvFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=ToWY0R45y8BHjmAdWxteY+oDuIkdik3lI8PSNGM4h2M=; b=ESuD7kfYCcl727sd3sZUURj4W70I6aJBmrS3RYt2wCn33kBnKlJzhvXlty4MW0mG5o cA7d0sqOtbCqJMZ2NUe0UcZAzDx8PaJ9oevEwqRWD+AIhLwTZJOcu6WnjXYsUYdST60/ vtlawi4WdX27sU9tzDOjQTqDGe7+Zkk57XAUsOJWmym4Ys+5vlmoZ76av40iv/7PNT79 V85Tynpgg0x95QlUKRqztOfp9QJM0erQBjdm9e9QnvRB/Az0O1E7FK+NTu3Hz3Kkzvlr MJ9J672HZeP0z4qCvnTJ4WgXAN2fsWwf0tqpzQFW6Bq3uvTLrYEE1s3j1jqPejIQ0zyM 2OWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="I/Ui5km2"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id m8si523769qte.391.2019.04.02.21.22.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:22:51 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="I/Ui5km2"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 6768821908; Wed, 3 Apr 2019 00:22:51 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:22:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=ToWY0R45y8BHjmAdWxteY+oDuIkdik3lI8PSNGM4h2M=; b=I/Ui5km2 fzUg9Gk9NdXhf11L7d39CpWXho85pgZB3EdjjtN9gRTTuxOVubTuvbpUmRBh84Ee TbLt9X12k2w2Ck5Y6ubD9qYTYuDf2PxVW/jlY37hiNQGUXii1W8TgHdYHuLQ5Kx8 2LPMvlFeI+bhIf54SrVprANJinvsUSTzHqMoOX6nf+H9qwfddHnwxfZhIxFcyAAo Zp9Ub5B3DOqc8g8MtltyXnY31R084eF33UB9lFNI+j0L0NRniqVo7Faort+zO+uh WLiWoR1QQCZWQaP9KSdayPpDRWjff/WRnkIDd3NNwLqAMaEVKqkJzUlbpj72zC8r m8YEo78EAkmftA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpedt X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 6A7E1100E5; Wed, 3 Apr 2019 00:22:44 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 01/14] slub: Add isolate() and migrate() methods Date: Wed, 3 Apr 2019 15:21:14 +1100 Message-Id: <20190403042127.18755-2-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the two methods needed for moving objects and enable the display of the callbacks via the /sys/kernel/slab interface. Add documentation explaining the use of these methods and the prototypes for slab.h. Add functions to setup the callbacks method for a slab cache. Add empty functions for SLAB/SLOB. The API is generic so it could be theoretically implemented for these allocators as well. Change sysfs 'ctor' field to be 'ops' to contain all the callback operations defined for a slab cache. Display the existing 'ctor' callback in the ops fields contents along with 'isolate' and 'migrate' callbacks. Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- include/linux/slab.h | 70 ++++++++++++++++++++++++++++++++++++++++ include/linux/slub_def.h | 3 ++ mm/slub.c | 59 +++++++++++++++++++++++++++++---- 3 files changed, 126 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..886fc130334d 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -154,6 +154,76 @@ void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); void memcg_destroy_kmem_caches(struct mem_cgroup *); +/* + * Function prototypes passed to kmem_cache_setup_mobility() to enable + * mobile objects and targeted reclaim in slab caches. + */ + +/** + * typedef kmem_cache_isolate_func - Object migration callback function. + * @s: The cache we are working on. + * @ptr: Pointer to an array of pointers to the objects to isolate. + * @nr: Number of objects in @ptr array. + * + * The purpose of kmem_cache_isolate_func() is to pin each object so that + * they cannot be freed until kmem_cache_migrate_func() has processed + * them. This may be accomplished by increasing the refcount or setting + * a flag. + * + * The object pointer array passed is also passed to + * kmem_cache_migrate_func(). The function may remove objects from the + * array by setting pointers to %NULL. This is useful if we can + * determine that an object is being freed because + * kmem_cache_isolate_func() was called when the subsystem was calling + * kmem_cache_free(). In that case it is not necessary to increase the + * refcount or specially mark the object because the release of the slab + * lock will lead to the immediate freeing of the object. + * + * Context: Called with locks held so that the slab objects cannot be + * freed. We are in an atomic context and no slab operations + * may be performed. + * Return: A pointer that is passed to the migrate function. If any + * objects cannot be touched at this point then the pointer may + * indicate a failure and then the migration function can simply + * remove the references that were already obtained. The private + * data could be used to track the objects that were already pinned. + */ +typedef void *kmem_cache_isolate_func(struct kmem_cache *s, void **ptr, int nr); + +/** + * typedef kmem_cache_migrate_func - Object migration callback function. + * @s: The cache we are working on. + * @ptr: Pointer to an array of pointers to the objects to migrate. + * @nr: Number of objects in @ptr array. + * @node: The NUMA node where the object should be allocated. + * @private: The pointer returned by kmem_cache_isolate_func(). + * + * This function is responsible for migrating objects. Typically, for + * each object in the input array you will want to allocate an new + * object, copy the original object, update any pointers, and free the + * old object. + * + * After this function returns all pointers to the old object should now + * point to the new object. + * + * Context: Called with no locks held and interrupts enabled. Sleeping + * is possible. Any operation may be performed. + */ +typedef void kmem_cache_migrate_func(struct kmem_cache *s, void **ptr, + int nr, int node, void *private); + +/* + * kmem_cache_setup_mobility() is used to setup callbacks for a slab cache. + */ +#ifdef CONFIG_SLUB +void kmem_cache_setup_mobility(struct kmem_cache *, kmem_cache_isolate_func, + kmem_cache_migrate_func); +#else +static inline void +kmem_cache_setup_mobility(struct kmem_cache *s, kmem_cache_isolate_func isolate, + kmem_cache_migrate_func migrate) {} +#endif + /* * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above. diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index d2153789bd9f..2879a2f5f8eb 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -99,6 +99,9 @@ struct kmem_cache { gfp_t allocflags; /* gfp flags to use on each alloc */ int refcount; /* Refcount for slab cache destroy */ void (*ctor)(void *); + kmem_cache_isolate_func *isolate; + kmem_cache_migrate_func *migrate; + unsigned int inuse; /* Offset to metadata */ unsigned int align; /* Alignment */ unsigned int red_left_pad; /* Left redzone padding size */ diff --git a/mm/slub.c b/mm/slub.c index d30ede89f4a6..ae44d640b8c1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4326,6 +4326,33 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return err; } +void kmem_cache_setup_mobility(struct kmem_cache *s, + kmem_cache_isolate_func isolate, + kmem_cache_migrate_func migrate) +{ + /* + * Mobile objects must have a ctor otherwise the object may be + * in an undefined state on allocation. Since the object may + * need to be inspected by the migration function at any time + * after allocation we must ensure that the object always has a + * defined state. + */ + if (!s->ctor) { + pr_err("%s: require constructor to setup mobility\n", s->name); + return; + } + + s->isolate = isolate; + s->migrate = migrate; + + /* + * Sadly serialization requirements currently mean that we have + * to disable fast cmpxchg based processing. + */ + s->flags &= ~__CMPXCHG_DOUBLE; +} +EXPORT_SYMBOL(kmem_cache_setup_mobility); + void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller) { struct kmem_cache *s; @@ -5010,13 +5037,33 @@ static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf, } SLAB_ATTR(cpu_partial); -static ssize_t ctor_show(struct kmem_cache *s, char *buf) +static int op_show(char *buf, const char *txt, unsigned long addr) { - if (!s->ctor) - return 0; - return sprintf(buf, "%pS\n", s->ctor); + int x = 0; + + x += sprintf(buf, "%s : ", txt); + x += sprint_symbol(buf + x, addr); + x += sprintf(buf + x, "\n"); + + return x; +} + +static ssize_t ops_show(struct kmem_cache *s, char *buf) +{ + int x = 0; + + if (s->ctor) + x += op_show(buf + x, "ctor", (unsigned long)s->ctor); + + if (s->isolate) + x += op_show(buf + x, "isolate", (unsigned long)s->isolate); + + if (s->migrate) + x += op_show(buf + x, "migrate", (unsigned long)s->migrate); + + return x; } -SLAB_ATTR_RO(ctor); +SLAB_ATTR_RO(ops); static ssize_t aliases_show(struct kmem_cache *s, char *buf) { @@ -5429,7 +5476,7 @@ static struct attribute *slab_attrs[] = { &objects_partial_attr.attr, &partial_attr.attr, &cpu_slabs_attr.attr, - &ctor_attr.attr, + &ops_attr.attr, &aliases_attr.attr, &align_attr.attr, &hwcache_align_attr.attr, From patchwork Wed Apr 3 04:21:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882743 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9EAC91575 for ; Wed, 3 Apr 2019 04:23:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 86862287A6 for ; Wed, 3 Apr 2019 04:23:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 79D1E289BF; Wed, 3 Apr 2019 04:23:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C8D54287A6 for ; Wed, 3 Apr 2019 04:23:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C38EC6B0272; Wed, 3 Apr 2019 00:23:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C12346B0274; Wed, 3 Apr 2019 00:23:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD8F86B0275; Wed, 3 Apr 2019 00:23:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 8CE346B0272 for ; Wed, 3 Apr 2019 00:23:00 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id v18so15706383qtk.5 for ; Tue, 02 Apr 2019 21:23:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=bxV5jBaXvA/LKL0qOR0ZLSK5IIfdrfWAd0nyjsopfiM=; b=LueBaAUI1iE6FSybqpbGnejtYEdauhGZARUzXb2j6OoTGJ5+P6poKaWG1FgIjNy+5t DJQAImHV2kHEgJyZm9IGMsZ6E29BMwm+Br6eRpvCVvJuZLpludW/owPq+egVfm5Goqc9 TOdGmZndOdwsSCulaX19JwzlE6Q8HszH3EAK8VyIiN5KzTm1hRVFh5EAh5sUxH+kXJLp ExIm1Y/WC3jdbc2/1ua78yJHSLc9B7CacUA45fO2lCb/MKQF7PokHFtH+U/y52hn4Amm PTcOWwecV1UkoamojKozIl1SSSLZbIS/GE1I1fJTn835EiGo0TgunjuDCNoHY2IF/n9w Ow/A== X-Gm-Message-State: APjAAAUqjoqUglO5NCvznIRA1WBFmEUYPGb55bdi4qReJnPxCfMLMwBI Q5jmc8A70GMVKsxTcsiwOqwTzhL17d833XAKzvBNmedsus6GiyeTI1k1L4vCARwGDPlzHSdiI/U Y0lcCIVFf/YE7hVDlOBbKsXfTf81zPNqxB1NBiWPrCqct/Q4b/rch0r7rCdTTF48= X-Received: by 2002:a0c:e587:: with SMTP id t7mr37965219qvm.114.1554265380319; Tue, 02 Apr 2019 21:23:00 -0700 (PDT) X-Google-Smtp-Source: APXvYqzoPrXPc1LRgN/bxCcXBhvShRz0TvCdJn+K5iARaPh1LfsG9lcJGtaJeo2rRuqR6I00BhPv X-Received: by 2002:a0c:e587:: with SMTP id t7mr37965175qvm.114.1554265379057; Tue, 02 Apr 2019 21:22:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265379; cv=none; d=google.com; s=arc-20160816; b=tjK8ROP8lzXRgPDs/8JG0vtu1+qBo0nvnmr38g1NJKsF0J5oR4ysOyoW+xBTAmZLXt r/tvz1XyvsJiwhr//uLXOz3r1oMVNiYaGlaM5RVSHhC5n0Iy3UH275lZDbwPQuut9H4y Ds0JLWaUCcTmEB2qys8BBoOFCPB4HhLbIXcdOHCFOG/uDxHCaxLktjimYb6JJ+mV719g 5K/S82ZvBz6Y7XyC6ZskFh5oCQ2GzGkgOUTH9QDZ5pvxrwYrSZpG3ej7UFDDaS9/wgTu sWliAiBGxOJTQmEz+/f5xTYW/graA6lNwr/bjTehbgd9LPW4fjzadsldyxnARwhjDTZY NU1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=bxV5jBaXvA/LKL0qOR0ZLSK5IIfdrfWAd0nyjsopfiM=; b=Av8ReY6h+lNYDEdG+wqljtjaDHy3bcaMkdmyfuQHZerfAm+kSEdOVMsb9R88Kcf11q pjeBHw3hhFVdWb2kga++cKbdjR+E0uzcsU4+m4ebpWTvj6FE69i7EQdf23mZzUCV8nu0 EYkHehj48GkimcRhhrFuOGTFjp2UYRJpy5JmWskyZmAfK8CGajL1bHPt9h9Pg1uPgW/0 5r/diwWPmdRlfG3/kPPXXBAyxKBzmWoHSCqwzK6EV8SKeSZLnddSbRNL96sDk9x1wkxz sTN26FQkzGGb2pFZ2OadjAdjiPHfj/hVKSlXMiLu/giyQdsg9tvs3PPw2u5e0lTkQSXh s2Ug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="V/Y3w93P"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id i12si2734380qtr.73.2019.04.02.21.22.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:22:59 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="V/Y3w93P"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id AE48921B24; Wed, 3 Apr 2019 00:22:58 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:22:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=bxV5jBaXvA/LKL0qOR0ZLSK5IIfdrfWAd0nyjsopfiM=; b=V/Y3w93P eDWWJ0zDKkgrTdnG9yUUrsY+karzjq+Fn9idovC3n7/bivCQ7R1bsVM2c5YGscG8 sR2mhr+En1765Mq5xj+9KQKGLCVvAOGhwa15dx6UudoCxQHsXMVVuFZP9u8Ur/Cg LN7WzigSvztiJ4vcOGkHnTyf2WFc8d0DGepG8WyyPxmN5Eiz2pPVh3dFmdbibVL6 6oapkwkiWiJzBt64IaSFbbr/wyJFPg2DqNMe9yE09bSc+nUCgRO8E5Bjq2kCcaIf H+CSus/Gw9OONyrpzMtxWCC6If1SeD25wsJ7SFu0lZzXvl6aAmSE7Jqb3XQAGmdN zUjMb6akz8gRCw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpedu X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id D33C210380; Wed, 3 Apr 2019 00:22:51 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 02/14] tools/vm/slabinfo: Add support for -C and -M options Date: Wed, 3 Apr 2019 15:21:15 +1100 Message-Id: <20190403042127.18755-3-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP -C lists caches that use a ctor. -M lists caches that support object migration. Add command line options to show caches with a constructor and caches that are movable (i.e. have migrate function). Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 40 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index 73818f1b2ef8..cbfc56c44c2f 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -33,6 +33,7 @@ struct slabinfo { unsigned int hwcache_align, object_size, objs_per_slab; unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; + int movable, ctor; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; unsigned long free_fastpath, free_slowpath; @@ -67,6 +68,8 @@ int show_report; int show_alias; int show_slab; int skip_zero = 1; +int show_movable; +int show_ctor; int show_numa; int show_track; int show_first_alias; @@ -109,11 +112,13 @@ static void fatal(const char *x, ...) static void usage(void) { - printf("slabinfo 4/15/2011. (c) 2007 sgi/(c) 2011 Linux Foundation.\n\n" - "slabinfo [-aADefhilnosrStTvz1LXBU] [N=K] [-dafzput] [slab-regexp]\n" + printf("slabinfo 4/15/2017. (c) 2007 sgi/(c) 2011 Linux Foundation/(c) 2017 Jump Trading LLC.\n\n" + "slabinfo [-aACDefhilMnosrStTvz1LXBU] [N=K] [-dafzput] [slab-regexp]\n" + "-a|--aliases Show aliases\n" "-A|--activity Most active slabs first\n" "-B|--Bytes Show size in bytes\n" + "-C|--ctor Show slabs with ctors\n" "-D|--display-active Switch line format to activity\n" "-e|--empty Show empty slabs\n" "-f|--first-alias Show first alias\n" @@ -121,6 +126,7 @@ static void usage(void) "-i|--inverted Inverted list\n" "-l|--slabs Show slabs\n" "-L|--Loss Sort by loss\n" + "-M|--movable Show caches that support movable objects\n" "-n|--numa Show NUMA information\n" "-N|--lines=K Show the first K slabs\n" "-o|--ops Show kmem_cache_ops\n" @@ -588,6 +594,12 @@ static void slabcache(struct slabinfo *s) if (show_empty && s->slabs) return; + if (show_ctor && !s->ctor) + return; + + if (show_movable && !s->movable) + return; + if (sort_loss == 0) store_size(size_str, slab_size(s)); else @@ -602,6 +614,10 @@ static void slabcache(struct slabinfo *s) *p++ = '*'; if (s->cache_dma) *p++ = 'd'; + if (s->ctor) + *p++ = 'C'; + if (s->movable) + *p++ = 'M'; if (s->hwcache_align) *p++ = 'A'; if (s->poison) @@ -636,7 +652,8 @@ static void slabcache(struct slabinfo *s) printf("%-21s %8ld %7d %15s %14s %4d %1d %3ld %3ld %s\n", s->name, s->objects, s->object_size, size_str, dist_str, s->objs_per_slab, s->order, - s->slabs ? (s->partial * 100) / s->slabs : 100, + s->slabs ? (s->partial * 100) / + (s->slabs * s->objs_per_slab) : 100, s->slabs ? (s->objects * s->object_size * 100) / (s->slabs * (page_size << s->order)) : 100, flags); @@ -1256,6 +1273,13 @@ static void read_slab_dir(void) slab->alloc_node_mismatch = get_obj("alloc_node_mismatch"); slab->deactivate_bypass = get_obj("deactivate_bypass"); chdir(".."); + if (read_slab_obj(slab, "ops")) { + if (strstr(buffer, "ctor :")) + slab->ctor = 1; + if (strstr(buffer, "migrate :")) + slab->movable = 1; + } + if (slab->name[0] == ':') alias_targets++; slab++; @@ -1332,6 +1356,8 @@ static void xtotals(void) } struct option opts[] = { + { "ctor", no_argument, NULL, 'C' }, + { "movable", no_argument, NULL, 'M' }, { "aliases", no_argument, NULL, 'a' }, { "activity", no_argument, NULL, 'A' }, { "debug", optional_argument, NULL, 'd' }, @@ -1367,7 +1393,7 @@ int main(int argc, char *argv[]) page_size = getpagesize(); - while ((c = getopt_long(argc, argv, "aAd::Defhil1noprstvzTSN:LXBU", + while ((c = getopt_long(argc, argv, "aACd::Defhil1MnoprstvzTSN:LXBU", opts, NULL)) != -1) switch (c) { case '1': @@ -1376,6 +1402,9 @@ int main(int argc, char *argv[]) case 'a': show_alias = 1; break; + case 'C': + show_ctor = 1; + break; case 'A': sort_active = 1; break; @@ -1399,6 +1428,9 @@ int main(int argc, char *argv[]) case 'i': show_inverted = 1; break; + case 'M': + show_movable = 1; + break; case 'n': show_numa = 1; break; From patchwork Wed Apr 3 04:21:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882745 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8A2B61575 for ; Wed, 3 Apr 2019 04:23:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 72C2C287A6 for ; Wed, 3 Apr 2019 04:23:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 66650289BF; Wed, 3 Apr 2019 04:23:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF2C0287A6 for ; Wed, 3 Apr 2019 04:23:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 004C56B0274; Wed, 3 Apr 2019 00:23:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id F1D9C6B0275; Wed, 3 Apr 2019 00:23:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E0F436B0276; Wed, 3 Apr 2019 00:23:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id C09156B0274 for ; Wed, 3 Apr 2019 00:23:07 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id p26so15549307qtq.21 for ; Tue, 02 Apr 2019 21:23:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=oAX6zCohmPiQ74kYACc2QgeImmemzleYDH4h+URdt/s=; b=ubdcPIROTKI7jbU6JOaJJ58d+zJBuOuLB0g+SZIGvXB25mtt5nv20UZkel7vZWvOSV Zdewn/P7oRGWTiYlNb2/Pyc5291o6zGqc275GXH93W0u6z4VUXrSW8lX9zmt6/fRJmvl trfvD6w87a2HobPInyFy4Czgvp3A869gmhNZgmDM/lU3d92QPaXr7s9jEbiDXQu2L+cC HzoGBCqxsakcGBMTwUFo4iZbzvhy0W0S9fX84LDfV6tNNLWuTKD65apsf1LCuMA5MIKI 9Uq4AUp0UsCGIcf5OLLeGxB371AlKBMuJRH7S0/6KoYhM8fwwsO8swgS6j3zcJZC6Z6D aF3g== X-Gm-Message-State: APjAAAW6UdtKi1GRfYQsGJpnrqD9TSl9NmJGlMtGdTm/GJx3LUV7gwRK 5NsVuE9i46Si8pnjR0I30iGvF1lG5kThAY7EMeO9k7+gnZWFDYLXWQJ18QkyFFmLvGrMh0/2uow eMSzr+sFdOaQQkTQKhSXyVLBQZ7ofsjFsYhBxD0dysoXg1fzo7iwrAtoIFh5rfGo= X-Received: by 2002:a0c:9319:: with SMTP id d25mr59129603qvd.99.1554265387520; Tue, 02 Apr 2019 21:23:07 -0700 (PDT) X-Google-Smtp-Source: APXvYqzWB3vAwe08xEj5XFAE6dhb6ugnyewSv4Bl5uRS6VbepGyxWONit/fuJQt3GGTkCJXKcIs6 X-Received: by 2002:a0c:9319:: with SMTP id d25mr59129570qvd.99.1554265386658; Tue, 02 Apr 2019 21:23:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265386; cv=none; d=google.com; s=arc-20160816; b=VZIEH1eV0sGWvquukL0flfqcQxDyX3Lc9SlW1CpU+5HDa5FMchG5GcIJK64I/WoTsj NJI59vlnhkpzLCYRTGBXxZgN5pIWRhuybD64aLqCEbtP9uO7xDLKYxpDHwx2KXCrnjLd jQZzQKnIkWDtofpUUppOevyTQUyJikZum/C/eCPI0K4TA6QnxWyqjdXyRVdlwMO73jKX ktaugaoBw4DQPPinty1RHc1hVWhSuuFC5elK6dA9IvGBkeJSyqYSIvPudlkNlC8/GiQl Y4xUEq+FP1VQGicEkT9HylzrIyBTSH3fy5HgyxTd+usgQGFEnw9DATaXgfTZ4CSGiBej NYgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=oAX6zCohmPiQ74kYACc2QgeImmemzleYDH4h+URdt/s=; b=ewyBZuAqL5ojx+g0RMNgaNteq9jvlpLwiPPjIPpE6ikkpkEUAYyf8ts/kIwWEdFW8x 4hrPujW6R68ndz0UnNuXHpsz6Uol6iMGpBLEEMQH/H7kWMQILLNsMkIwd2SO0qqancaW dP7JZ0jSxD1ruy+esBBCS2/SIodxoOWqKBhk94fhfzi5Bk07NLSU/a73QjoIkzY551Rd cukwLGxd0ObSejO8eGTZuRYU8sxR1h44fZg8GgRtQnZFARncsg6QM30v7hFK3Ac0wUuX NaSAHhn1te7wctLCnrQjRoLrQ3mxlU9fyxwVJFu+kWyMVR0cYwuZoLJYxL0VO1hTSN4q 4XUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=FC9CRiNy; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id y6si342406qkf.93.2019.04.02.21.23.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:23:06 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=FC9CRiNy; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 57E9821B2A; Wed, 3 Apr 2019 00:23:06 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:23:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=oAX6zCohmPiQ74kYACc2QgeImmemzleYDH4h+URdt/s=; b=FC9CRiNy /A2QACAxPg4m3dmOCUdhhqa5feWxuaX7P0DN1cHtAGYWp1mGWP79PKIKnCk61N8b HJ+e0qEBWAWBOZ8aIklqcS6HduIcqSzhR2lHIemxwotBmC58QMnVTGS32LM3UIt+ 6qp1Xg1tEinuIHZtRcvmbN2JKESdUCciDey+5yqaPvwCkJ3TRgHkUAnA+MXn7aKv JIremYOdC/D8Zo9iexYBt+p91801gl7QHym/o0JuDzUYmYJMhU+7p0rN4tFvmNGt qdleqeBuW4FoFEBbAUdh3EXUMNNhPut1FvkVUfwRAj9AenIa0OO01aO3RbaLsYPo fSbc5pRf/Vpe6w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpedv X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 43DB310310; Wed, 3 Apr 2019 00:22:59 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 03/14] slub: Sort slab cache list Date: Wed, 3 Apr 2019 15:21:16 +1100 Message-Id: <20190403042127.18755-4-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It is advantageous to have all defragmentable slabs together at the beginning of the list of slabs so that there is no need to scan the complete list. Put defragmentable caches first when adding a slab cache and others last. Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- mm/slab_common.c | 2 +- mm/slub.c | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..db5e9a0b1535 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -393,7 +393,7 @@ static struct kmem_cache *create_cache(const char *name, goto out_free_cache; s->refcount = 1; - list_add(&s->list, &slab_caches); + list_add_tail(&s->list, &slab_caches); memcg_link_cache(s); out: if (err) diff --git a/mm/slub.c b/mm/slub.c index ae44d640b8c1..f6b0e4a395ef 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4342,6 +4342,8 @@ void kmem_cache_setup_mobility(struct kmem_cache *s, return; } + mutex_lock(&slab_mutex); + s->isolate = isolate; s->migrate = migrate; @@ -4350,6 +4352,10 @@ void kmem_cache_setup_mobility(struct kmem_cache *s, * to disable fast cmpxchg based processing. */ s->flags &= ~__CMPXCHG_DOUBLE; + + list_move(&s->list, &slab_caches); /* Move to top */ + + mutex_unlock(&slab_mutex); } EXPORT_SYMBOL(kmem_cache_setup_mobility); From patchwork Wed Apr 3 04:21:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882747 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B28271575 for ; Wed, 3 Apr 2019 04:23:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9723B287A6 for ; Wed, 3 Apr 2019 04:23:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8B0BB289BF; Wed, 3 Apr 2019 04:23:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 30D66287A6 for ; Wed, 3 Apr 2019 04:23:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 314746B0275; Wed, 3 Apr 2019 00:23:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2E80B6B0276; Wed, 3 Apr 2019 00:23:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1B1336B0277; Wed, 3 Apr 2019 00:23:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id E9E186B0275 for ; Wed, 3 Apr 2019 00:23:15 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id 77so13639761qkd.9 for ; Tue, 02 Apr 2019 21:23:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=7KCR/bG3UyviFO0P3mTQEntswZ73LUL6n099eGv7Rtc=; b=Z5bW8F6xyNaukiQXK0jlrdJS+k7HjxR0I8R89ZTtWtuD7UtjNNG6ljUgp+n/Ee6E6e unqo/vYQrF8y9sTvVtkBj4qtu6JDZKjhh3SSCiyjLKgnZYWt4+LQhFNZ8Cx/ez4LNkrj 5oFlBV/BY4xqiLd2W4TvSSjQBbVwKjdsUoKOQ+Z6RNpYvSfTQ2k7VQ9HGLck3cfpGHCl D93VRXk/FlRKgf6vU7crDrjgdmX5YldKWTSTsCPixmaPlm5dI1RtTnLb6nmCdQjsY+jk MNWm2Dr/9ciZqh0qdLSlX2TWP5DhDCpOiXVIMRt5tMZrgyF9ROhCeTyweHeE/Dd2+S3K QRhQ== X-Gm-Message-State: APjAAAWG6BEHGNdfrWKRGjmazKn0AvNcXIEfLwunSjcOo0sDu+uUseaO o9sd7fH5M9UZRVjwZ84W+iNyS645bC/1zb+VGJXC97BqyqYywwp7fvubbjecZuZUEotBDtIkI/V 9bEP1/bOHbHhoQtXqmNxL+MB6zDHgbgN8tgbhK39UgWrYZp1qJ4hkkle1GbpMFbA= X-Received: by 2002:ac8:85c:: with SMTP id x28mr62745391qth.90.1554265395594; Tue, 02 Apr 2019 21:23:15 -0700 (PDT) X-Google-Smtp-Source: APXvYqyG26NdXqKA0U5CiIbSHf/VMTOj9Rk+zaAtCxLaQssIPMyr6CE0T5djibWCZG+twW/YlpOL X-Received: by 2002:ac8:85c:: with SMTP id x28mr62745330qth.90.1554265394037; Tue, 02 Apr 2019 21:23:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265394; cv=none; d=google.com; s=arc-20160816; b=lPEUbifkD4WnKTj2jS9ZcB7/+HxeW0KcnFVgB/Geo5qSUoPhrVXR1T59JSDVjFWwYe 5Xm61WDnizxs7i/Q3HbA/Waf1s1uljDZWn1BVSnaGUs9A59Yl9FXlHOYt7UhB+GV9MA7 /bqOIcTwS6fOKyj6vx5RJsQciIATpdxMTxs1Jc5G5hhdMeHGBDFgoGVFoW4ufKJpDu3T kjqaxExLJYVZxbUPMSw/qL6GApzVRTooHNQmtU5YgUt65TP9HHZl3iEkHboj9YDuCTKh kOkfda+/S079QwVZp2KzOqLeAt/zH39AdzwJ5kSLN6KqnEaRV2p+bB9GPLGvJV0dpKVM tr2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=7KCR/bG3UyviFO0P3mTQEntswZ73LUL6n099eGv7Rtc=; b=YwgwhXL1vneeKTcf5zjWko3hYR9sLMXlA6c+0k3ekMNk7Cah6u1ZEgbjNxbzEbX88Y gRTckt3wB8eKlzHbcZojuDdb/j8V9XAIB9R+9uSZQtd3gN16dSjFeyPIVJVKjEer/QQ/ Lc05hbKlCdryUAp7OJPS+C8rZDTUp64EGJ4BLi6RrvcylAxrd6Ak89nEiJMb3anl14py 6kc2/K04hwzoTdxulaF5NCwBUY0JSq+n/xXp7zKa0Rq/URYGpcKgOcyk/QM21R7dPTp4 2/NuW9iH7Nc8ZxN1DUQYM4mjF6G5W331iNuDiAo0OJszIGIbosszwCE1iok4j3EdwgfL qNQw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ax3tkqcr; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id l49si8884522qtk.159.2019.04.02.21.23.13 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:23:14 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ax3tkqcr; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id BFAF521B62; Wed, 3 Apr 2019 00:23:13 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:23:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=7KCR/bG3UyviFO0P3mTQEntswZ73LUL6n099eGv7Rtc=; b=ax3tkqcr iE4NZNVHdVUciAmuuOOB8VSlg3Mk0WNyGBebOUJrR1kFO5Sz44KarnJn7qrwZ++e Q2UdjGlbCqPXOQmABUNXts7ejIpcJLxZNWmMffBQvHmsUlfkMy7a58iVl/sOBEYd JXmWe8juapFbQkEwIGyAAxMjgccuNUZ2TeX7rxHU3bRVdVjQFgXY0lLt+3pzqCJU Uvfa+rRg71shL9MJOqzAVv93sqKmMzcBhlYyX2H2HG/wUBBicTfnHO1V3PhFINhH +qsabvZ3pcJTzvx26z0/F4yWtJCJTBV37iCSxzuqWmcD1cXhaxhvoGDFqDVFt27Z OsPqMol0Oi4P1A== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeef X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id B3851100E5; Wed, 3 Apr 2019 00:23:06 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 04/14] slub: Slab defrag core Date: Wed, 3 Apr 2019 15:21:17 +1100 Message-Id: <20190403042127.18755-5-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Internal fragmentation can occur within pages used by the slub allocator. Under some workloads large numbers of pages can be used by partial slab pages. This under-utilisation is bad simply because it wastes memory but also because if the system is under memory pressure higher order allocations may become difficult to satisfy. If we can defrag slab caches we can alleviate these problems. Implement Slab Movable Objects in order to defragment slab caches. Slab defragmentation may occur: 1. Unconditionally when __kmem_cache_shrink() is called on a slab cache by the kernel calling kmem_cache_shrink(). 2. Unconditionally through the use of the slabinfo command. slabinfo -s 3. Conditionally via the use of kmem_cache_defrag() - Use Slab Movable Objects when shrinking cache. Currently when the kernel calls kmem_cache_shrink() we curate the partial slabs list. If object migration is not enabled for the cache we still do this, if however, SMO is enabled we attempt to move objects in partially full slabs in order to defragment the cache. Shrink attempts to move all objects in order to reduce the cache to a single partial slab for each node. - Add conditional per node defrag via new function: kmem_defrag_slabs(int node). kmem_defrag_slabs() attempts to defragment all slab caches for node. Defragmentation is done conditionally dependent on MAX_PARTIAL _AND_ defrag_used_ratio. Caches are only considered for defragmentation if the number of partial slabs exceeds MAX_PARTIAL (per node). Also, defragmentation only occurs if the usage ratio of the slab is lower than the configured percentage (sysfs field added in this patch). Fragmentation ratios are measured by calculating the percentage of objects in use compared to the total number of objects that the slab page can accommodate. The scanning of slab caches is optimized because the defragmentable slabs come first on the list. Thus we can terminate scans on the first slab encountered that does not support defragmentation. kmem_defrag_slabs() takes a node parameter. This can either be -1 if defragmentation should be performed on all nodes, or a node number. Defragmentation may be disabled by setting defrag ratio to 0 echo 0 > /sys/kernel/slab//defrag_used_ratio - Add a defrag ratio sysfs field and set it to 30% by default. A limit of 30% specifies that more than 3 out of 10 available slots for objects need to be in use otherwise slab defragmentation will be attempted on the remaining objects. In order for a cache to be defragmentable the cache must support object migration (SMO). Enabling SMO for a cache is done via a call to the recently added function: void kmem_cache_setup_mobility(struct kmem_cache *, kmem_cache_isolate_func, kmem_cache_migrate_func); Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- Documentation/ABI/testing/sysfs-kernel-slab | 14 + include/linux/slab.h | 1 + include/linux/slub_def.h | 7 + mm/slub.c | 385 ++++++++++++++++---- 4 files changed, 334 insertions(+), 73 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-kernel-slab b/Documentation/ABI/testing/sysfs-kernel-slab index 29601d93a1c2..7770c03be6b4 100644 --- a/Documentation/ABI/testing/sysfs-kernel-slab +++ b/Documentation/ABI/testing/sysfs-kernel-slab @@ -180,6 +180,20 @@ Description: list. It can be written to clear the current count. Available when CONFIG_SLUB_STATS is enabled. +What: /sys/kernel/slab/cache/defrag_used_ratio +Date: February 2019 +KernelVersion: 5.0 +Contact: Christoph Lameter + Pekka Enberg , +Description: + The defrag_used_ratio file allows the control of how aggressive + slab fragmentation reduction works at reclaiming objects from + sparsely populated slabs. This is a percentage. If a slab has + less than this percentage of objects allocated then reclaim will + attempt to reclaim objects so that the whole slab page can be + freed. 0% specifies no reclaim attempt (defrag disabled), 100% + specifies attempt to reclaim all pages. The default is 30%. + What: /sys/kernel/slab/cache/deactivate_to_tail Date: February 2008 KernelVersion: 2.6.25 diff --git a/include/linux/slab.h b/include/linux/slab.h index 886fc130334d..4bf381b34829 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -149,6 +149,7 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, void (*ctor)(void *)); void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); +unsigned long kmem_defrag_slabs(int node); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index 2879a2f5f8eb..34c6f1250652 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -107,6 +107,13 @@ struct kmem_cache { unsigned int red_left_pad; /* Left redzone padding size */ const char *name; /* Name (only for display!) */ struct list_head list; /* List of slab caches */ + int defrag_used_ratio; /* + * Ratio used to check against the + * percentage of objects allocated in a + * slab page. If less than this ratio + * is allocated then reclaim attempts + * are made. + */ #ifdef CONFIG_SYSFS struct kobject kobj; /* For sysfs */ struct work_struct kobj_remove_work; diff --git a/mm/slub.c b/mm/slub.c index f6b0e4a395ef..e601c804ed79 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -354,6 +354,12 @@ static __always_inline void slab_lock(struct page *page) bit_spin_lock(PG_locked, &page->flags); } +static __always_inline int slab_trylock(struct page *page) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + return bit_spin_trylock(PG_locked, &page->flags); +} + static __always_inline void slab_unlock(struct page *page) { VM_BUG_ON_PAGE(PageTail(page), page); @@ -3643,6 +3649,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) set_cpu_partial(s); + s->defrag_used_ratio = 30; #ifdef CONFIG_NUMA s->remote_node_defrag_ratio = 1000; #endif @@ -3959,79 +3966,6 @@ void kfree(const void *x) } EXPORT_SYMBOL(kfree); -#define SHRINK_PROMOTE_MAX 32 - -/* - * kmem_cache_shrink discards empty slabs and promotes the slabs filled - * up most to the head of the partial lists. New allocations will then - * fill those up and thus they can be removed from the partial lists. - * - * The slabs with the least items are placed last. This results in them - * being allocated from last increasing the chance that the last objects - * are freed in them. - */ -int __kmem_cache_shrink(struct kmem_cache *s) -{ - int node; - int i; - struct kmem_cache_node *n; - struct page *page; - struct page *t; - struct list_head discard; - struct list_head promote[SHRINK_PROMOTE_MAX]; - unsigned long flags; - int ret = 0; - - flush_all(s); - for_each_kmem_cache_node(s, node, n) { - INIT_LIST_HEAD(&discard); - for (i = 0; i < SHRINK_PROMOTE_MAX; i++) - INIT_LIST_HEAD(promote + i); - - spin_lock_irqsave(&n->list_lock, flags); - - /* - * Build lists of slabs to discard or promote. - * - * Note that concurrent frees may occur while we hold the - * list_lock. page->inuse here is the upper limit. - */ - list_for_each_entry_safe(page, t, &n->partial, lru) { - int free = page->objects - page->inuse; - - /* Do not reread page->inuse */ - barrier(); - - /* We do not keep full slabs on the list */ - BUG_ON(free <= 0); - - if (free == page->objects) { - list_move(&page->lru, &discard); - n->nr_partial--; - } else if (free <= SHRINK_PROMOTE_MAX) - list_move(&page->lru, promote + free - 1); - } - - /* - * Promote the slabs filled up most to the head of the - * partial list. - */ - for (i = SHRINK_PROMOTE_MAX - 1; i >= 0; i--) - list_splice(promote + i, &n->partial); - - spin_unlock_irqrestore(&n->list_lock, flags); - - /* Release empty slabs */ - list_for_each_entry_safe(page, t, &discard, lru) - discard_slab(s, page); - - if (slabs_node(s, node)) - ret = 1; - } - - return ret; -} - #ifdef CONFIG_MEMCG static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) { @@ -4326,6 +4260,287 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return err; } +/* + * Allocate a slab scratch space that is sufficient to keep pointers to + * individual objects for all objects in cache and also a bitmap for the + * objects (used to mark which objects are active). + */ +static inline void *alloc_scratch(struct kmem_cache *s) +{ + unsigned int size = oo_objects(s->max); + + return kmalloc(size * sizeof(void *) + + BITS_TO_LONGS(size) * sizeof(unsigned long), + GFP_KERNEL); +} + +/* + * move_slab_page() - Move all objects in the given slab. + * @page: The slab we are working on. + * @scratch: Pointer to scratch space. + * @node: The target node to move objects to. + * + * If the target node is not the current node then the object is moved + * to the target node. If the target node is the current node then this + * is an effective way of defragmentation since the current slab page + * with its object is exempt from allocation. + */ +static void move_slab_page(struct page *page, void *scratch, int node) +{ + unsigned long objects; + struct kmem_cache *s; + unsigned long flags; + unsigned long *map; + void *private; + int count; + void *p; + void **vector = scratch; + void *addr = page_address(page); + + local_irq_save(flags); + slab_lock(page); + + BUG_ON(!PageSlab(page)); /* Must be a slab page */ + BUG_ON(!page->frozen); /* Slab must have been frozen earlier */ + + s = page->slab_cache; + objects = page->objects; + map = scratch + objects * sizeof(void **); + + /* Determine used objects */ + bitmap_fill(map, objects); + for (p = page->freelist; p; p = get_freepointer(s, p)) + __clear_bit(slab_index(p, s, addr), map); + + /* Build vector of pointers to objects */ + count = 0; + memset(vector, 0, objects * sizeof(void **)); + for_each_object(p, s, addr, objects) + if (test_bit(slab_index(p, s, addr), map)) + vector[count++] = p; + + if (s->isolate) + private = s->isolate(s, vector, count); + else + /* Objects do not need to be isolated */ + private = NULL; + + /* + * Pinned the objects. Now we can drop the slab lock. The slab + * is frozen so it cannot vanish from under us nor will + * allocations be performed on the slab. However, unlocking the + * slab will allow concurrent slab_frees to proceed. So the + * subsystem must have a way to tell from the content of the + * object that it was freed. + * + * If neither RCU nor ctor is being used then the object may be + * modified by the allocator after being freed which may disrupt + * the ability of the migrate function to tell if the object is + * free or not. + */ + slab_unlock(page); + local_irq_restore(flags); + + /* Perform callback to move the objects */ + s->migrate(s, vector, count, node, private); +} + +/* + * kmem_cache_defrag() - Defragment node. + * @s: cache we are working on. + * @node: The node to move objects from. + * @target_node: The node to move objects to. + * @ratio: The defrag ratio (percentage, between 0 and 100). + * + * Release slabs with zero objects and try to call the migration function + * for slabs with less than the 'ratio' percentage of objects allocated. + * + * Moved objects are allocated on @target_node. + * + * Return: The number of partial slabs left on @node after the + * operation. + */ +static unsigned long kmem_cache_defrag(struct kmem_cache *s, + int node, int target_node, int ratio) +{ + struct kmem_cache_node *n = get_node(s, node); + struct page *page, *page2; + LIST_HEAD(move_list); + unsigned long flags; + + if (node == target_node && n->nr_partial <= 1) { + /* + * Trying to reduce fragmentation on a node but there is + * only a single or no partial slab page. This is already + * the optimal object density that we can reach. + */ + return n->nr_partial; + } + + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &n->partial, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + if (page->inuse) { + if (page->inuse > ratio * page->objects / 100) { + slab_unlock(page); + /* + * Skip slab because the object density + * in the slab page is high enough. + */ + continue; + } + + list_move(&page->lru, &move_list); + if (s->migrate) { + /* Stop page being considered for allocations */ + n->nr_partial--; + page->frozen = 1; + } + slab_unlock(page); + } else { /* Empty slab page */ + list_del(&page->lru); + n->nr_partial--; + slab_unlock(page); + discard_slab(s, page); + } + } + + if (!s->migrate) { + /* + * No defrag method. By simply putting the zaplist at + * the end of the partial list we can let them simmer + * longer and thus increase the chance of all objects + * being reclaimed. + */ + list_splice(&move_list, n->partial.prev); + } + + spin_unlock_irqrestore(&n->list_lock, flags); + + if (s->migrate && !list_empty(&move_list)) { + void **scratch = alloc_scratch(s); + if (scratch) { + /* Try to remove / move the objects left */ + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + } + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + /* + * Objects left in slab page, move it to the + * tail of the partial list to increase the + * chance that the freeing of the remaining + * objects will free the slab page. + */ + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); + } + + return n->nr_partial; +} + +/** + * kmem_defrag_slabs() - Defrag slab caches. + * @node: The node to defrag or -1 for all nodes. + * + * Defrag slabs conditional on the amount of fragmentation in a page. + * + * Return: The total number of partial slabs in migratable caches left + * on @node after the operation. + */ +unsigned long kmem_defrag_slabs(int node) +{ + struct kmem_cache *s; + unsigned long left = 0; + int nid; + + if (node >= MAX_NUMNODES) + return -EINVAL; + + /* + * kmem_defrag_slabs() may be called from the reclaim path which + * may be called for any page allocator alloc. So there is the + * danger that we get called in a situation where slub already + * acquired the slub_lock for other purposes. + */ + if (!mutex_trylock(&slab_mutex)) + return 0; + + list_for_each_entry(s, &slab_caches, list) { + /* + * Defragmentable caches come first. If the slab cache is + * not defragmentable then we can stop traversing the list. + */ + if (!s->migrate) + break; + + if (node >= 0) { + if (s->node[node]->nr_partial > MAX_PARTIAL) { + left += kmem_cache_defrag(s, node, node, + s->defrag_used_ratio); + } + continue; + } + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (s->node[nid]->nr_partial > MAX_PARTIAL) { + left += kmem_cache_defrag(s, nid, nid, + s->defrag_used_ratio); + } + } + } + mutex_unlock(&slab_mutex); + return left; +} +EXPORT_SYMBOL(kmem_defrag_slabs); + +/** + * __kmem_cache_shrink() - Shrink a cache. + * @s: The cache to shrink. + * + * Reduces the memory footprint of a slab cache by as much as possible. + * + * This works by: + * 1. Removing empty slabs from the partial list. + * 2. Migrating slab objects to denser slab pages if the slab cache + * supports migration. If not, reorganizing the partial list so that + * more densely allocated slab pages come first. + * + * Not called directly, called by kmem_cache_shrink(). + */ +int __kmem_cache_shrink(struct kmem_cache *s) +{ + int node; + int left = 0; + + flush_all(s); + for_each_node_state(node, N_NORMAL_MEMORY) + left += kmem_cache_defrag(s, node, node, 100); + + return left; +} +EXPORT_SYMBOL(__kmem_cache_shrink); + void kmem_cache_setup_mobility(struct kmem_cache *s, kmem_cache_isolate_func isolate, kmem_cache_migrate_func migrate) @@ -5177,6 +5392,29 @@ static ssize_t destroy_by_rcu_show(struct kmem_cache *s, char *buf) } SLAB_ATTR_RO(destroy_by_rcu); +static ssize_t defrag_used_ratio_show(struct kmem_cache *s, char *buf) +{ + return sprintf(buf, "%d\n", s->defrag_used_ratio); +} + +static ssize_t defrag_used_ratio_store(struct kmem_cache *s, + const char *buf, size_t length) +{ + unsigned long ratio; + int err; + + err = kstrtoul(buf, 10, &ratio); + if (err) + return err; + + if (ratio > 100) + return -EINVAL; + + s->defrag_used_ratio = ratio; + return length; +} +SLAB_ATTR(defrag_used_ratio); + #ifdef CONFIG_SLUB_DEBUG static ssize_t slabs_show(struct kmem_cache *s, char *buf) { @@ -5501,6 +5739,7 @@ static struct attribute *slab_attrs[] = { &validate_attr.attr, &alloc_calls_attr.attr, &free_calls_attr.attr, + &defrag_used_ratio_attr.attr, #endif #ifdef CONFIG_ZONE_DMA &cache_dma_attr.attr, From patchwork Wed Apr 3 04:21:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882749 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3136617E1 for ; Wed, 3 Apr 2019 04:23:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B55A289DA for ; Wed, 3 Apr 2019 04:23:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0F65B289DD; Wed, 3 Apr 2019 04:23:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9421B289DA for ; Wed, 3 Apr 2019 04:23:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 92B996B0276; Wed, 3 Apr 2019 00:23:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8DC186B0277; Wed, 3 Apr 2019 00:23:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D5DF6B0278; Wed, 3 Apr 2019 00:23:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id 5FE3D6B0276 for ; Wed, 3 Apr 2019 00:23:23 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id g7so13544181qkb.7 for ; Tue, 02 Apr 2019 21:23:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=B8+D4dqa1BX+6QU8onGZrdHDeV/bBw1xA9Qf7WlA766fW2OULQXpl+p/EEfMDGE1QX Pqua13g2MiJQ6j8JI5+W/H4KKqRAi1fGjp8o1uThw1Hk98BvMBAjk1tY7+K4Nb2ONpo/ 583iGCBGhBAD/FrCupuWg23xf7qMljgOEoLWBrhsycypZsGZxdiMDNh/6YyxWwMYORPS XdA8nwqgWcBcTs0vHnNKNCl26jFbMYI228zK7yl854fNwn+dUOhp4LltjfPVaJg1pcZz jZIsO4ky7is82GpFHjotqdOlCvRDgjEGhZUmOee0qJLKidm0My8AGr5pybatH4/Stpat 4H/A== X-Gm-Message-State: APjAAAXB1Fqu6OjUMH/WbyJ8G3X6oIvaGsHDHLzK2pZ/rer1Gi8Prqsh L/PmfBnjSM2OvusjbfES5eNQyf7+Ucm1s+dhUDL2c5eUMIHkW0DgQiShSYxtEROC054uEQ1250V Iz02op2vZ8vHnyMWMvbE50IZgeDZNDotdj8twW9SjhjcwvdWqY00inhiL7jVNAZg= X-Received: by 2002:a05:620a:14b4:: with SMTP id x20mr45357811qkj.202.1554265403115; Tue, 02 Apr 2019 21:23:23 -0700 (PDT) X-Google-Smtp-Source: APXvYqyPQCjT2t3f2KMGBVFNCCOuZeKsMx1u4QWopDE5p4OLLryPyjiTQyOpslIjBQRU+Zt+h+c0 X-Received: by 2002:a05:620a:14b4:: with SMTP id x20mr45357772qkj.202.1554265402186; Tue, 02 Apr 2019 21:23:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265402; cv=none; d=google.com; s=arc-20160816; b=fn0cWP7/tqUG0Me6McaeUbrFswssJwTclcAxUDd5hmkF2HiEwVRa+r3H6Gg9jU3gkE n7CcEYGBkyhDsMeaeWRKlEC3tdzUmXOq/u7mGp1VKPE3g/V3vXAU1mFShc6XgXflHJEL KVWUzKTR81bPCCnBGWSI90J63ufaxE8az1HkwjMtadov2suAX+H5vqqY0URmIH5+ud1e Qz6ItP79Y9CKeGIwc1MVJ9Apkn24DVpjvSFPWUFKSipV8LUb5VvRndjDbIPc3koNiomx jM16GyuwCyPEAa0r0ypiwHIxwPKSjHQc50Ds/blry7O8UMB2ZANizOWw7wEcYYDUPN3q eY0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=xkZj7+Zc12qTR1LxmT2emSgfOBEoKNwnGy1x8etDYX5O8GE3NLqRCKvBgMeJ21Mm+2 6vM4Vu3/5DIUNf5GeYM1hHRDPjhftvCktfWnKrzEF+8maq8JQfxEvIn/89hGQMWyabyf 9L077AyEs/9mqZAInXYLa1B/oqeaG1fVfxttOyJFfBrFaymP9bl7GR1G3HKaQFFu6k9n owvBtbNP6aX9zLzvVHfvoSwiWsELjt1w12ssD9B/RDgXuM6Xy7VEAIj0wLqPzwS46A63 2cbBckvodOF1AIvEampORKW81ZqltcHTTFC/YicEOqEf/Eqjh1CupeYz0hky4f5jfJyD uXjg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=mTY6Adzl; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id k12si8807886qth.145.2019.04.02.21.23.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:23:22 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=mTY6Adzl; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id D808121F01; Wed, 3 Apr 2019 00:23:21 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:23:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=mTY6Adzl 61j/HX4/u8pVRmeDhbPrC5pgYVGektr0Ftsd0SvT2OiwWBG1APy9EJGh9fp48NnB EkghxIwUbm2vc5eGdfY6Ri7B1WZ7dFE83Q8KjoMLauPYrqMiBsUm7PfnTkFi1GJ7 AMMY5EutgtgXsI8Nvo/RteIagRpivu78gCb49FFY6+o+kHO1NKoRklIllmfpv7kv SmuTEtJ/ShiCV989LSfZZTOhLMZ4ZXsflOqw1ZbkGS2U53evb8jOl/5M8KsC23VZ L9h53+Lc1hmENNQehrHIUgnv+MsJH0r7Z+r+aKzB9oSnGpEq2VNsYTck9ArRnnLM jI2WmSnybJjAnw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeeg X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 5D044100E5; Wed, 3 Apr 2019 00:23:14 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 05/14] tools/vm/slabinfo: Add remote node defrag ratio output Date: Wed, 3 Apr 2019 15:21:18 +1100 Message-Id: <20190403042127.18755-6-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add output line for NUMA remote node defrag ratio. Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index cbfc56c44c2f..d2c22f9ee2d8 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -34,6 +34,7 @@ struct slabinfo { unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; int movable, ctor; + int remote_node_defrag_ratio; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; unsigned long free_fastpath, free_slowpath; @@ -377,6 +378,10 @@ static void slab_numa(struct slabinfo *s, int mode) if (skip_zero && !s->slabs) return; + if (mode) { + printf("\nNUMA remote node defrag ratio: %3d\n", + s->remote_node_defrag_ratio); + } if (!line) { printf("\n%-21s:", mode ? "NUMA nodes" : "Slab"); for(node = 0; node <= highest_node; node++) @@ -1272,6 +1277,8 @@ static void read_slab_dir(void) slab->cpu_partial_free = get_obj("cpu_partial_free"); slab->alloc_node_mismatch = get_obj("alloc_node_mismatch"); slab->deactivate_bypass = get_obj("deactivate_bypass"); + slab->remote_node_defrag_ratio = + get_obj("remote_node_defrag_ratio"); chdir(".."); if (read_slab_obj(slab, "ops")) { if (strstr(buffer, "ctor :")) From patchwork Wed Apr 3 04:21:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882751 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 88EF517E1 for ; Wed, 3 Apr 2019 04:23:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 727D6289B5 for ; Wed, 3 Apr 2019 04:23:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 63D8E289C7; Wed, 3 Apr 2019 04:23:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5998289B5 for ; Wed, 3 Apr 2019 04:23:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D98846B0277; Wed, 3 Apr 2019 00:23:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D70CB6B0278; Wed, 3 Apr 2019 00:23:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C5F5D6B0279; Wed, 3 Apr 2019 00:23:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id A2B136B0277 for ; Wed, 3 Apr 2019 00:23:30 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id f196so13643321qke.4 for ; Tue, 02 Apr 2019 21:23:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=b+5gwRgWFajPZy+qqDmBe1rjk+9Uc+oaPmN0A6LFLL4Zk49w7uudPp7ee/EC7HVAM+ B1MNVSHAGCzqZKQhX2lQCX9entNcTnn4LE4lZQhRWMvWBgXJc1sKuW8RXdKae/8WuHiJ H/PuomiYwOA36mI1tTWL6RcTHMUMHswe0PiRuFDn1tNWLH+1akLMxcdcq9RUDL3NaIg0 j3wViUnbb4kawUpRg0Z+P11FFaIna8VZ/F9KaDLFJWK0i4cfu+5aSt8BSow7INewB9Ad fnTMkGZszuLBbIMZUMjVoEYJcNnQZRmwUJNnZcVYCKnfOr+RRH4T25zlyQoQHygVjMPp FQEQ== X-Gm-Message-State: APjAAAWMytVjuolq3gtlLD2dHo3EG+tZC+glGSu2eIVTl+2JYYutVQuq G+wUschzOmsT6BD6g+DivymzDzPQNdw+kzRIkjrPV5Dhryl/tZz6DyPfc9YLrnMAzOVSpbuzlMQ 519g8JkC1YUo8ZeVz4hOqvk+PMxLZpD8YEmrMkWlUrXsFhRLGJ84aYJet644qnKA= X-Received: by 2002:a05:620a:15b5:: with SMTP id f21mr59624713qkk.89.1554265410429; Tue, 02 Apr 2019 21:23:30 -0700 (PDT) X-Google-Smtp-Source: APXvYqyfvbwUyS0/IOSpSOaAsr08piEMqkTD1olWF7ZsupKagphkhVXsqvZUo2jUXkM6s7cxIY5u X-Received: by 2002:a05:620a:15b5:: with SMTP id f21mr59624679qkk.89.1554265409585; Tue, 02 Apr 2019 21:23:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265409; cv=none; d=google.com; s=arc-20160816; b=xDG5k3ddvOwCW3WoaqjXaPG6T2Ybn+fbhkWAg8sQeRfdnbnyFPsZZZQ+Zlgely+DeS Q0hFWuVi+OCaHi/xfOqVhpRs3hbSZeloYSIr6qDDX7DGm+BpDvTQs33FNaHER19M+YZl BraNQunTQI1KkpyeMO+txkQFZbgDDWOccmDYXmZ2l0oKusbcivap+62AMCBCA+sA76OH +LGW0ROc6+BvjHJYkuNuOYfHxIJov2NBxuFBJeRVosp+psTimWadcMAHfeUNF34EQFhv FZeC0B50B7RFdJZ/wCfsFfv4kiJ9btR21zOhz4aVMwKfrEsJvOquyAhuWmtyalemQSQT qvKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=ueETmMaPJkY9ld9mEru/iU+xXdjPYYaGI1Ra1sOjG/e7t50cl6Ds6c0H/fOIYTqrSI 0RNvLccpQFsrahGrrVbG/uL/cFmckHcUlyJM5AaxwPwCib50YW2or6sxUDZmGLIlkPuC WW78mJLFVDHsUdA+FMvQVPxZk4rFOTC1AipCu1dB3zNQzN2eFD2AWtnJJ+g6SUxJnHXl kAyfn17uGoKrM/Ck1X9venpiwxSBb5mLlsTumONkDGHi74vxrA6WD77buOX+4r2wUAZC cRznCOsFB3Jb/E4jEx7lUM8yFnZ+tS39oslDzhafjx2abR9PizrRTHgDVfD1vKubEpmG KYOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=8g49H2KY; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id u16si2645531qka.158.2019.04.02.21.23.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:23:29 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=8g49H2KY; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 4DE0121F26; Wed, 3 Apr 2019 00:23:29 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:23:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=8g49H2KY JPyh5+opdBo9WW/hQaD63ziek5gLUqZ1SJAEGX/UPnwhMGr6q5qStSpPv8+QQxjR POc2MSuSFcOpQ9JZVhJXiQ+CsQsAJYapK45RsWHMctm3CZYuKSE6kMSi70ZPCWOX sDJ6uFRFKyVqXJFxXTeB78uOR1Uoq647tk4/PcINYiihTDwM4BZa9nbT5K8/3Glz PMRpIDbhDh91Nf5jKli946Lb0KAegvU1nuuPBHNX6VPf5srmtyz6QdzcdHCEzha6 gEZZuskbhtHXFyoOlPvr6z/b3XwxrmRNq2QxqohPy8bPTBFDQFZw1MDSApW4a04l vCd/yIyM8PLOQw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeeh X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 5D5D5100E5; Wed, 3 Apr 2019 00:23:22 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 06/14] tools/vm/slabinfo: Add defrag_used_ratio output Date: Wed, 3 Apr 2019 15:21:19 +1100 Message-Id: <20190403042127.18755-7-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add output for the newly added defrag_used_ratio sysfs knob. Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index d2c22f9ee2d8..ef4ff93df4cc 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -34,6 +34,7 @@ struct slabinfo { unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; int movable, ctor; + int defrag_used_ratio; int remote_node_defrag_ratio; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; @@ -549,6 +550,8 @@ static void report(struct slabinfo *s) printf("** Slabs are destroyed via RCU\n"); if (s->reclaim_account) printf("** Reclaim accounting active\n"); + if (s->movable) + printf("** Defragmentation at %d%%\n", s->defrag_used_ratio); printf("\nSizes (bytes) Slabs Debug Memory\n"); printf("------------------------------------------------------------------------\n"); @@ -1279,6 +1282,7 @@ static void read_slab_dir(void) slab->deactivate_bypass = get_obj("deactivate_bypass"); slab->remote_node_defrag_ratio = get_obj("remote_node_defrag_ratio"); + slab->defrag_used_ratio = get_obj("defrag_used_ratio"); chdir(".."); if (read_slab_obj(slab, "ops")) { if (strstr(buffer, "ctor :")) From patchwork Wed Apr 3 04:21:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882753 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1D0B71575 for ; Wed, 3 Apr 2019 04:23:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 03039287A6 for ; Wed, 3 Apr 2019 04:23:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA893289BF; Wed, 3 Apr 2019 04:23:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 75E5C287A6 for ; Wed, 3 Apr 2019 04:23:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 583F06B0278; Wed, 3 Apr 2019 00:23:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 559946B0279; Wed, 3 Apr 2019 00:23:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3FA956B027A; Wed, 3 Apr 2019 00:23:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 1B75A6B0278 for ; Wed, 3 Apr 2019 00:23:40 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id 77so13640451qkd.9 for ; Tue, 02 Apr 2019 21:23:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=cA1XOCXkbNUUGn3mtT3hWjbalquBVtxLn4JS1PbP8tn0NYAgedYrQQtOEJY6bsToRm Q1g+YBLz9brZd3g3CB7LciYZ1kxK6/vkWIWsp74ZKJAKIjnEA+wf69ek5lAAb/CkjKHl 28bpOgxNwZDtIzsj/mrE4xvTVsEgBmc1NzxvYFtSgiQCHHjOHxEO2EXFRVpdNcnJkpV+ Ae43kswsnfFDYE8iZwgm08vhkOMMmtGScJrrMhJ1r4eg4tb1wJophz9nvcn+iC/Edsne D2d6Re1aGduPYq4ChdRiMUryyZrRJBUQS4rYT87YhMqM5qxIKZ0LSi4OTr8x7DNifNF0 pK/w== X-Gm-Message-State: APjAAAVtWDGlTu/bZBdftzkgsBoQXZ8Obj1dSZgZvDOkJPRo3NXpQ8RX HXFgJCsW+O9WgcvWg4KPuFbBv+yflqrB8eIv15CeDONJIIWELxLTGw5Dx9vK8Wfj533Igrr9LBD O3xx//ku/4XscR1xSQGlzlNm/Gqz4equce0e1k0O2tzo7YnrSk/gdQkIYJHAND4Q= X-Received: by 2002:ac8:234e:: with SMTP id b14mr62406038qtb.271.1554265419799; Tue, 02 Apr 2019 21:23:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqzYhxzLtheTDHFJQTadwV2jl6P5hNM3Gh3p8mPq518hBVnstXO5lonxsq7hjRTHxuMm60Ds X-Received: by 2002:ac8:234e:: with SMTP id b14mr62405959qtb.271.1554265417513; Tue, 02 Apr 2019 21:23:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265417; cv=none; d=google.com; s=arc-20160816; b=OXQkVKn85xhTYVeEa7PhTvKGFei+iZ4dtfXaf87Zphg5sgDJfkr3h5GR6iB7YxH2mT azbbi9Jy2MtZ193FqMyD+MxpkEaGXcKHdU/9zWOVdINVBy5IMsgkI9+OgCfw8DP8JWxE x1QwZbEraKi2D41KBMb67ddqz3EJ6LtKf+lIoUI1P4WLd5otbrmiDtY5nhtwWqhJgeR4 RneLbCERUM9snk2aPFHAVYWtJOmQtz5bT/NQA44ZDHxwnI+rNFGwpGuwzbBeTgSp/G3D KKiU8Dl3ZE5bqBYBIc+58Qu6+Skz3RG0x9XPUpbeLVag4MNcYCabMGzI7XyAbBwBuB0v JiOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=DQx7S/Lk96WiBWLkx5qHhyFCbemkYTW/Dd0XOg9NcqxbXmnM0jkk3o1s9Q2QGcASRF lrJeN1hSrVpP/xrddIZunwEViAirKL2q5jhPIvriuN+OTk5S6aFNpDWr4Ov3ZQETayIo GIjfrbpZvIjuFfNkeYRBXDAkQcIcO32NOGBl4+Zf9Bw0qsMIke2ZE5blyQqR0WVV39iM n/2jz1yTGkDN6XNBUkmfpaWN2quhNwg/LmuTwwgK4UgMidUpjjdYLUA1fQ613XXxTolv WJZB1WkvzNRLdhyj+Njd/aNJ3j1j2aI9rP0qH3CUsHp9AATgEY96oPYWJ8Sc3blVuVAq lUug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="hfeG2c/B"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id w7si1072679qki.216.2019.04.02.21.23.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:23:37 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="hfeG2c/B"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 37C6621F3D; Wed, 3 Apr 2019 00:23:37 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:23:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=hfeG2c/B xzYjJLAQ/cAK9V/bm5sf9OelZn4StOa++RHf7JFsH0EfY1Mxn28ziO/IaxKqU60D VKWVhhVr7Gsgn/wnXUp27syiMEiEj1rgQcRx4tL+jWB2YRCtvejgEgDVgWWJcUKz UtBReLcPGcTzea8B+Rq+3pnpB0SMmzsYclUJVtoU31Mf3tjGvRhtrN+c5nhWtKh0 BYD7fek08RJF9W4QCGJfzkwozw8ajNIa6UTQC05Z5cnUYM71tmBdL6WI79CkwuIj VqjypdNeftRXra78jQ6dAqBwiAGOqmaMYsNfJKYviOl5OKqUlAVGB1deBdHCjM6j QdS7UNZ8J3IZuQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeei X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id C0734100E5; Wed, 3 Apr 2019 00:23:29 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 07/14] tools/testing/slab: Add object migration test module Date: Wed, 3 Apr 2019 15:21:20 +1100 Message-Id: <20190403042127.18755-8-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just implemented slab movable objects for the SLUB allocator. We should test that code. In order to do so we need to be able to do a number of things - Create a cache - Enable Slab Movable Objects for the cache - Allocate objects to the cache - Free objects from within specific slabs of the cache We can do all this via a loadable module. Add a module that defines functions that can be triggered from userspace via a debugfs entry. From the source: /* * SLUB defragmentation a.k.a. Slab Movable Objects (SMO). * * This module is used for testing the SLUB allocator. Enables * userspace to run kernel functions via a debugfs file. * * debugfs: /sys/kernel/debugfs/smo/callfn (write only) * * String written to `callfn` is parsed by the module and associated * function is called. See fn_tab for mapping of strings to functions. */ References to allocated objects are kept by the module in a linked list so that userspace can control which object to free. We introduce the following four functions via the function table "enable": Enables object migration for the test cache. "alloc X": Allocates X objects "free X [Y]": Frees X objects starting at list position Y (default Y==0) "test": Runs [stress] tests from within the module (see below). {"enable", smo_enable_cache_mobility}, {"alloc", smo_alloc_objects}, {"free", smo_free_object}, {"test", smo_run_module_tests}, Freeing from the start of the list creates a hole in the slab being freed from (i.e. creates a partial slab). The results of running these commands can be see using `slabinfo` (available in tools/vm/): make -o slabinfo tools/vm/slabinfo.c Stress tests can be run from within the module. These tests are internal to the module because we verify that object references are still good after object migration. These are called 'stress' tests because it is intended that they create/free a lot of objects. Userspace can control the number of objects to create, default is 1000. Example test session -------------------- Relevant /proc/slabinfo column headers: name # mount -t debugfs none /sys/kernel/debug/ $ cd path/to/linux/tools/testing/slab; make ... # insmod slub_defrag.ko # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 0 0 392 20 2 From this we can see that the module created cache 'smo_test' with 20 objects per slab and 2 pages per slab (and cache is currently empty). We can play with the slab allocator manually: # insmod slub_defrag.ko # echo 'alloc 21' > callfn # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 21 40 392 20 2 We see here that 21 active objects have been allocated creating 2 slabs (40 total objects). # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 21 Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 2 Sanity Checks : On Total: 16384 SlabObj: 392 Full : 1 Redzoning : On Used : 1176 SlabSiz: 8192 Partial: 1 Poisoning : On Loss : 15208 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 7056 Align : 8 Objects: 20 Tracing : Off Lpadd: 704 Now free an object from the first slot of the first slab # echo 'free 1' > callfn # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 20 40 392 20 2 # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 20 Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 2 Sanity Checks : On Total: 16384 SlabObj: 392 Full : 0 Redzoning : On Used : 1120 SlabSiz: 8192 Partial: 2 Poisoning : On Loss : 15264 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 6720 Align : 8 Objects: 20 Tracing : Off Lpadd: 704 Calling shrink now on the cache does nothing because object migration is not enabled (output omitted). If we enable object migration then shrink the cache we expect the object from the second slab to me moved to the first slot in the first slab and the second slab to be removed from the partial list. # echo 'enable' > callfn # slabinfo smo_test --shrink # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 20 ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 1 Sanity Checks : On Total: 8192 SlabObj: 392 Full : 1 Redzoning : On Used : 1120 SlabSiz: 8192 Partial: 0 Poisoning : On Loss : 7072 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 6720 Align : 8 Objects: 20 Tracing : Off Lpadd: 352 We can run the stress tests (with the default number of objects): # cd /sys/kernel/debug/smo # echo 'test' > callfn [ 3.576617] smo: test using nr_objs: 1000 keep: 10 [ 3.580169] smo: Module tests completed successfully Signed-off-by: Tobin C. Harding --- tools/testing/slab/Makefile | 10 + tools/testing/slab/slub_defrag.c | 566 +++++++++++++++++++++++++++++++ 2 files changed, 576 insertions(+) create mode 100644 tools/testing/slab/Makefile create mode 100644 tools/testing/slab/slub_defrag.c diff --git a/tools/testing/slab/Makefile b/tools/testing/slab/Makefile new file mode 100644 index 000000000000..440c2e3e356f --- /dev/null +++ b/tools/testing/slab/Makefile @@ -0,0 +1,10 @@ +obj-m += slub_defrag.o + +KTREE=../../.. + +all: + make -C ${KTREE} M=$(PWD) modules + +clean: + make -C ${KTREE} M=$(PWD) clean + diff --git a/tools/testing/slab/slub_defrag.c b/tools/testing/slab/slub_defrag.c new file mode 100644 index 000000000000..4a5c24394b96 --- /dev/null +++ b/tools/testing/slab/slub_defrag.c @@ -0,0 +1,566 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * SLUB defragmentation a.k.a. Slab Movable Objects (SMO). + * + * This module is used for testing the SLUB allocator. Enables + * userspace to run kernel functions via a debugfs file. + * + * debugfs: /sys/kernel/debugfs/smo/callfn (write only) + * + * String written to `callfn` is parsed by the module and associated + * function is called. See fn_tab for mapping of strings to functions. + */ + +/* debugfs commands accept two optional arguments */ +#define SMO_CMD_DEFAUT_ARG -1 + +#define SMO_DEBUGFS_DIR "smo" +struct dentry *smo_debugfs_root; + +#define SMO_CACHE_NAME "smo_test" +static struct kmem_cache *cachep; + +struct smo_slub_object { + struct list_head list; + char buf[32]; /* Unused except to control size of object */ + long id; +}; + +/* Our list of allocated objects */ +LIST_HEAD(objects); + +static void list_add_to_objects(struct smo_slub_object *so) +{ + /* + * We free from the front of the list so store at the + * tail in order to put holes in the cache when we free. + */ + list_add_tail(&so->list, &objects); +} + +/** + * smo_object_ctor() - SMO object constructor function. + * @ptr: Pointer to memory where the object should be constructed. + */ +void smo_object_ctor(void *ptr) +{ + struct smo_slub_object *so = ptr; + + INIT_LIST_HEAD(&so->list); + memset(so->buf, 0, sizeof(so->buf)); + so->id = -1; +} + +/** + * smo_cache_migrate() - kmem_cache migrate function. + * @cp: kmem_cache pointer. + * @objs: Array of pointers to objects to migrate. + * @size: Number of objects in @objs. + * @node: NUMA node where the object should be allocated. + * @private: Pointer returned by kmem_cache_isolate_func(). + */ +void smo_cache_migrate(struct kmem_cache *cp, void **objs, int size, + int node, void *private) +{ + struct smo_slub_object **so_objs = (struct smo_slub_object **)objs; + struct smo_slub_object *so_old, *so_new; + int i; + + for (i = 0; i < size; i++) { + so_old = so_objs[i]; + + so_new = kmem_cache_alloc_node(cachep, GFP_KERNEL, node); + if (!so_new) { + pr_debug("kmem_cache_alloc failed\n"); + return; + } + + /* Copy object */ + so_new->id = so_old->id; + + /* Update references to old object */ + list_del(&so_old->list); + list_add_to_objects(so_new); + + kmem_cache_free(cachep, so_old); + } +} + +static int smo_enable_cache_mobility(int _unused, int __unused) +{ + /* Enable movable objects: BOOM! */ + kmem_cache_setup_mobility(cachep, NULL, smo_cache_migrate); + pr_info("smo: kmem_cache %s defrag enabled\n", SMO_CACHE_NAME); + return 0; +} + +/* + * smo_alloc_objects() - Allocate objects and store reference. + * @nr_objs: Number of objects to allocate. + * @node: NUMA node to allocate objects on. + * + * Allocates @n smo_slub_objects. Stores a reference to them in + * the global list of objects (at the tail of the list). + * + * Return: The number of objects allocated. + */ +static int smo_alloc_objects(int nr_objs, int node) +{ + struct smo_slub_object *so; + int i; + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1; + if (node == SMO_CMD_DEFAUT_ARG) + node = NUMA_NO_NODE; + + for (i = 0; i < nr_objs; i++) { + if (node == NUMA_NO_NODE) + so = kmem_cache_alloc(cachep, GFP_KERNEL); + else + so = kmem_cache_alloc_node(cachep, GFP_KERNEL, node); + if (!so) { + pr_err("smo: Failed to alloc object %d of %d\n", i, nr_objs); + return i; + } + list_add_to_objects(so); + } + return nr_objs; +} + +/* + * smo_free_object() - Frees n objects from position. + * @nr_objs: Number of objects to free. + * @pos: Position in global list to start freeing. + * + * Iterates over the global list of objects to position @pos then frees @n + * objects from there (or to end of list). Does nothing if @n > list length. + * + * Calling with @n==0 frees all objects starting at @pos. + * + * Return: Number of objects freed. + */ +static int smo_free_object(int nr_objs, int pos) +{ + struct smo_slub_object *cur, *tmp; + int deleted = 0; + int i = 0; + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1; + if (pos == SMO_CMD_DEFAUT_ARG) + pos = 0; + + list_for_each_entry_safe(cur, tmp, &objects, list) { + if (i < pos) { + i++; + continue; + } + + list_del(&cur->list); + kmem_cache_free(cachep, cur); + deleted++; + if (deleted == nr_objs) + break; + } + return deleted; +} + +static int index_for_expected_id(long *expected, int size, long id) +{ + int i; + + /* Array is unsorted, just iterate the whole thing */ + for (i = 0; i < size; i++) { + if (expected[i] == id) + return i; + } + return -1; /* Not found */ +} + +static int assert_have_objects(int nr_objs, int keep) +{ + struct smo_slub_object *cur; + long *expected; /* Array of expected IDs */ + int nr_ids; /* Length of array */ + long id; + int index, i; + + nr_ids = nr_objs / keep + 1; + + expected = kmalloc_array(nr_ids, sizeof(long), GFP_KERNEL); + if (!expected) + return -ENOMEM; + + id = 0; + for (i = 0; i < nr_ids; i++) { + expected[i] = id; + id += keep; + } + + list_for_each_entry(cur, &objects, list) { + index = index_for_expected_id(expected, nr_ids, cur->id); + if (index < 0) { + pr_err("smo: ID not found: %ld\n", cur->id); + return -1; + } + + if (expected[index] == -1) { + pr_err("smo: ID already encountered: %ld\n", cur->id); + return -1; + } + expected[index] = -1; + } + return 0; +} + +/* + * smo_run_module_tests() - Runs unit tests from within the module + * @nr_objs: Number of objects to allocate. + * @keep: Free all but 1 in @keep objects. + * + * Allocates @nr_objects then iterates over the allocated objects + * freeing all but 1 out of every @keep objects i.e. for @keep==10 + * keeps the first object then frees the next 9. + * + * Caller is responsible for ensuring that the cache has at most a + * single slab on the partial list without any objects in it. This is + * easy enough to ensure, just call this when the module is freshly + * loaded. + */ +static int smo_run_module_tests(int nr_objs, int keep) +{ + struct smo_slub_object *so; + struct smo_slub_object *cur, *tmp; + long i; + + if (!list_empty(&objects)) { + pr_err("smo: test requires clean module state\n"); + return -1; + } + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1000; + if (keep == SMO_CMD_DEFAUT_ARG) + keep = 10; + + pr_info("smo: test using nr_objs: %d keep: %d\n", nr_objs, keep); + + /* Perhaps we got called like this 'test 1000' */ + if (keep == 0) { + pr_err("Usage: test \n"); + return -1; + } + + /* Test constructor */ + so = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!so) { + pr_err("smo: Failed to alloc object\n"); + return -1; + } + if (so->id != -1) { + pr_err("smo: Initial state incorrect"); + return -1; + } + kmem_cache_free(cachep, so); + + /* + * Test that object migration is correctly implemented by module + * + * This gives us confidence that if new code correctly enables + * object migration (via correct implementation of migrate and + * isolate functions) then the slub allocator code that does + * object migration is correct. + */ + + for (i = 0; i < nr_objs; i++) { + so = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!so) { + pr_err("smo: Failed to alloc object %ld of %d\n", + i, nr_objs); + return -1; + } + so->id = (long)i; + list_add_to_objects(so); + } + + assert_have_objects(nr_objs, 1); + + i = 0; + list_for_each_entry_safe(cur, tmp, &objects, list) { + if (i++ % keep == 0) + continue; + + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + + /* Verify shrink does nothing when migration is not enabled */ + kmem_cache_shrink(cachep); + assert_have_objects(nr_objs, 1); + + /* Now test shrink */ + kmem_cache_setup_mobility(cachep, NULL, smo_cache_migrate); + kmem_cache_shrink(cachep); + /* + * Because of how migrate function deletes and adds objects to + * the objects list we have no way of knowing the order. We + * want to confirm that we have all the objects after shrink + * that we had before we did the shrink. + */ + assert_have_objects(nr_objs, keep); + + /* cleanup */ + list_for_each_entry_safe(cur, tmp, &objects, list) { + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + kmem_cache_shrink(cachep); /* Remove empty slabs from partial list */ + + pr_info("smo: Module tests completed successfully\n"); + return 0; +} + +/* + * struct functions() - Map command to a function pointer. + */ +struct functions { + char *fn_name; + int (*fn_ptr)(int arg0, int arg1); +} fn_tab[] = { + /* + * Because of the way we parse the function table no command + * may have another command as its prefix. + * i.e. this will break: 'foo' and 'foobar' + */ + {"enable", smo_enable_cache_mobility}, + {"alloc", smo_alloc_objects}, + {"free", smo_free_object}, + {"test", smo_run_module_tests}, +}; + +#define FN_TAB_SIZE (sizeof(fn_tab) / sizeof(struct functions)) + +/* + * parse_cmd_buf() - Gets command and arguments command string. + * @buf: Buffer containing the command string. + * @cmd: Out parameter, pointer to the command. + * @arg1: Out parameter, stores the first argument. + * @arg2: Out parameter, stores the second argument. + * + * Parses and tokenizes the input command buffer. Stores a pointer to the + * command (start of @buf) in @cmd. Stores the converted long values for + * argument 1 and 2 in the respective out parameters @arg1 and @arg2. + * + * Since arguments are optional, if they are not found the default values are + * returned. In order for the caller to differentiate defaults from arguments + * of the same value the number of arguments parsed is returned. + * + * Return: Number of arguments found. + */ +static int parse_cmd_buf(char *buf, char **cmd, long *arg1, long *arg2) +{ + int found; + char *ptr; + int ret; + + *arg1 = SMO_CMD_DEFAUT_ARG; + *arg2 = SMO_CMD_DEFAUT_ARG; + found = 0; + + /* Jump over the command, check if there are any args */ + ptr = strsep(&buf, " "); + if (!ptr || !buf) + return found; + + ptr = strsep(&buf, " "); + ret = kstrtol(ptr, 10, arg1); + if (ret < 0) { + pr_err("failed to convert arg, defaulting to %d. (%s)\n", + SMO_CMD_DEFAUT_ARG, ptr); + return found; + } + found++; + if (!buf) /* No second arg */ + return found; + + ptr = strsep(&buf, " "); + ret = kstrtol(ptr, 10, arg2); + if (ret < 0) { + pr_err("failed to convert arg, defaulting to %d. (%s)\n", + SMO_CMD_DEFAUT_ARG, ptr); + return found; + } + found++; + + return found; +} + +/* + * call_function() - Calls the function described by str. + * @str: ' []' + * + * Does table lookup on , calls appropriate function passing + * as a the argument. Optional arg defaults to 1. + */ +static void call_function(char *str) +{ + char *cmd; + long arg1 = 0; + long arg2 = 0; + int i; + + if (!str) + return; + + (void)parse_cmd_buf(str, &cmd, &arg1, &arg2); + + for (i = 0; i < FN_TAB_SIZE; i++) { + char *fn_name = fn_tab[i].fn_name; + + if (strcmp(fn_name, str) == 0) { + fn_tab[i].fn_ptr(arg1, arg2); + return; /* All done */ + } + } + + pr_err("failed to call function for cmd: %s\n", str); +} + +/* + * smo_callfn_debugfs_write() - debugfs write function. + * @file: User file + * @user_buf: Userspace buffer + * @len: Length of the user space buffer + * @off: Offset within the file + * + * Used for triggering functions by writing command to debugfs file. + * + * echo ' ' > /sys/kernel/debug/smo/callfn + * + * Return: Number of bytes copied if request succeeds, + * the corresponding error code otherwise. + */ +static ssize_t smo_callfn_debugfs_write(struct file *file, + const char __user *ubuf, + size_t len, + loff_t *off) +{ + char *kbuf; + int nbytes = 0; + + if (*off != 0 || len == 0) + return -EINVAL; + + kbuf = kzalloc(len, GFP_KERNEL); + if (!kbuf) + return -ENOMEM; + + nbytes = strncpy_from_user(kbuf, ubuf, len); + if (nbytes < 0) + goto out; + + if (kbuf[nbytes - 1] == '\n') + kbuf[nbytes - 1] = '\0'; + + call_function(kbuf); /* Tokenizes kbuf */ +out: + kfree(kbuf); + return nbytes; +} + +const struct file_operations fops_callfn_debugfs = { + .owner = THIS_MODULE, + .write = smo_callfn_debugfs_write, +}; + +static int __init smo_debugfs_init(void) +{ + struct dentry *d; + + smo_debugfs_root = debugfs_create_dir(SMO_DEBUGFS_DIR, NULL); + d = debugfs_create_file("callfn", 0200, smo_debugfs_root, NULL, + &fops_callfn_debugfs); + if (IS_ERR(d)) + return PTR_ERR(d); + + return 0; +} + +static void __exit smo_debugfs_cleanup(void) +{ + debugfs_remove_recursive(smo_debugfs_root); +} + +static int __init smo_cache_init(void) +{ + cachep = kmem_cache_create(SMO_CACHE_NAME, + sizeof(struct smo_slub_object), + 0, 0, smo_object_ctor); + if (!cachep) + return -1; + + return 0; +} + +static void __exit smo_cache_cleanup(void) +{ + struct smo_slub_object *cur, *tmp; + + list_for_each_entry_safe(cur, tmp, &objects, list) { + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + kmem_cache_destroy(cachep); +} + +static int __init smo_init(void) +{ + int ret; + + ret = smo_cache_init(); + if (ret) { + pr_err("smo: Failed to create cache\n"); + return ret; + } + pr_info("smo: Created kmem_cache: %s\n", SMO_CACHE_NAME); + + ret = smo_debugfs_init(); + if (ret) { + pr_err("smo: Failed to init debugfs\n"); + return ret; + } + pr_info("smo: Created debugfs directory: /sys/kernel/debugfs/%s\n", + SMO_DEBUGFS_DIR); + + pr_info("smo: Test module loaded\n"); + return 0; +} +module_init(smo_init); + +static void __exit smo_exit(void) +{ + smo_debugfs_cleanup(); + smo_cache_cleanup(); + + pr_info("smo: Test module removed\n"); +} +module_exit(smo_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Tobin C. Harding"); +MODULE_DESCRIPTION("SLUB Movable Objects test module."); From patchwork Wed Apr 3 04:21:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882755 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C2BC117E1 for ; Wed, 3 Apr 2019 04:23:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A938F287A6 for ; Wed, 3 Apr 2019 04:23:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9D36B289BF; Wed, 3 Apr 2019 04:23:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 77EB6287A6 for ; Wed, 3 Apr 2019 04:23:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 66DDE6B0279; Wed, 3 Apr 2019 00:23:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6434C6B027A; Wed, 3 Apr 2019 00:23:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 50B126B027B; Wed, 3 Apr 2019 00:23:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 2D9576B0279 for ; Wed, 3 Apr 2019 00:23:48 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id t22so15690283qtc.13 for ; Tue, 02 Apr 2019 21:23:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=JpRwUr4rao+yXJd3vXFs1OWDXMY/PLNCUUWLXqeczITyvOrlT09ZUSyKgDIjFQWlip 3im8L6SsJJ7kH44XL+WGH6W2m3WpzAy8a+AGsFfbpwywOWLVPOz+O9SuhWMNKjUl0ZNO Nu9OIouTNp7P3f3GhlCZW/ENJbZ/hxxFuG1VNT0d8R+5cOh7oIclcCNdoqLWTMdZbh6g BEifKPX6LYtgMlyx4BpLItVV7tow2zCY1bwBJWvZqyGzO7C8KLI5jUy/kkDOHRIpG5VB YdwACK0GgER44YV2u295EA76zkSEwAosniXOIY0Myu3rYCUCoL3JmcC1HcPJxtVWr4qm diXQ== X-Gm-Message-State: APjAAAX/pD7NJdwCfn9f4i6M+zdyG4rLsh3jGUMckcugVpxlDAbN5vYf j1J9e7LD6m4ip5yAIG2HU0jY7AGfAR3EtzmF5WUwJLRydnJSyBRxxRp86I4MO73UW2YcnJetl8F 75DMzmqLxNPMLCkqRYhePJIibOQ6Bk3hVElxzV6fKwML5CqH5dLOOY6GIUpH4eUM= X-Received: by 2002:a0c:d849:: with SMTP id i9mr60591616qvj.207.1554265427878; Tue, 02 Apr 2019 21:23:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqyNRi7xTUJsaW74uO2Wi5s6i+9UKpaKTY1HJ+n6gt9g796pePYveomznyJlI8XKCXk2syxF X-Received: by 2002:a0c:d849:: with SMTP id i9mr60591527qvj.207.1554265425456; Tue, 02 Apr 2019 21:23:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265425; cv=none; d=google.com; s=arc-20160816; b=dKt6iq9n6+RKSxge7ANFkz9sI9Bt62/MXL/M4zfo3CJb0uRSzWWdlx2IXyc8QsR17+ Id98Q18lktM9B4VBDWFjneNm2MWt6O9T3vijU6a6RNCfoi23gD2CZWulBj8d4CrMKP1n WqHbZ0lPX3bgzJsTrdX/YVgEzYfARw3ev0GxyS7PZ/voIbqSzhwsEtkuOqC0KRGeHimb wrhC+RaezIzsZO2Mg6qhvGPHvW7GeXE0Z0K2fOdqX1eK66FeAgxjGbkSSwuv22C1n4o1 Tdf/x6Knum/FfT2YiZwwaHlfEd/7UCh2+QOa45FvuFrUdW5E7Hefc7RuIxpd3x35N+0J 9U7A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=k82IBKyGpuT78sRoXX9Ht+7Duls8hugyl0si8MYXGA6gl3fCyhYLMBnFViRGW9hU3A kEJDX/nD0TaW5h+TjqU+RqfBaiTmlHG4Os5e4wi+5c1GwlAOpPiuc2enedj5dZjGSQvF e7RZN/jVVDL6K1ZJ95ySD50xcBIQB7rJYNZx7v/EI7A8GMLusaHe6o0H4VtvRIlxS2aV YilHj10X43DIUCzwewMHMgCAXATXYYcUolwK4kexXscNypOqVJjL16QZAXJJAXJgvg+r iZbozTNqPMUT7amB0DCn+12DqcM4xhkl/coSLmppz/H6QcGtwNlvnD05lBe+9EZMTXoX CJhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=CbWMx4VE; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id d16si2237104qtj.301.2019.04.02.21.23.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:23:45 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=CbWMx4VE; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 2FE8721F3C; Wed, 3 Apr 2019 00:23:45 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:23:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=CbWMx4VE CCohJL+6LL6weHupFLBvYOMVq6IYExGAPQUSRwnC5UPJ2sbjliDJ+xb3RGQzeNK5 HGqDtMbWuqqKbq5GGL29BNSNqBy8b/GP0RBneNsAKX7q09EdyKvtDcOKYf9mJ6di Qk7jEbVesQzjvdT8lLxzussc+ETh5lQ1oIZ0Z5vFOJyhluoGjz1Q9H1I2ZEM7zHg 8SIX4yBIPEmDyGq418/rVeQatra+D6nvxR2+lCknZNkZ1cMVcOrwg55t88m9Mw8O P0TC6NzRUbEVFB33HF6L+8BWQMTk+X596H+BrRQbUGUo2RWHVX8yPSKxUVlsIJRe ZOvYYyqjn4/w2w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeej X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id AAB37100E5; Wed, 3 Apr 2019 00:23:37 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 08/14] tools/testing/slab: Add object migration test suite Date: Wed, 3 Apr 2019 15:21:21 +1100 Message-Id: <20190403042127.18755-9-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just added a module that enables testing the SLUB allocators ability to defrag/shrink caches via movable objects. Tests are better when they are automated. Add automated testing via a python script for SLUB movable objects. Example output: $ cd path/to/linux/tools/testing/slab $ /slub_defrag.py Please run script as root $ sudo ./slub_defrag.py $ sudo ./slub_defrag.py --debug Loading module ... Slab cache smo_test created Objects per slab: 20 Running sanity checks ... Running module stress test (see dmesg for additional test output) ... Removing module slub_defrag ... Loading module ... Slab cache smo_test created Running test non-movable ... testing slab 'smo_test' prior to enabling movable objects ... verified non-movable slabs are NOT shrinkable Running test movable ... testing slab 'smo_test' after enabling movable objects ... verified movable slabs are shrinkable Removing module slub_defrag ... Signed-off-by: Tobin C. Harding --- tools/testing/slab/slub_defrag.c | 1 + tools/testing/slab/slub_defrag.py | 451 ++++++++++++++++++++++++++++++ 2 files changed, 452 insertions(+) create mode 100755 tools/testing/slab/slub_defrag.py diff --git a/tools/testing/slab/slub_defrag.c b/tools/testing/slab/slub_defrag.c index 4a5c24394b96..8332e69ee868 100644 --- a/tools/testing/slab/slub_defrag.c +++ b/tools/testing/slab/slub_defrag.c @@ -337,6 +337,7 @@ static int smo_run_module_tests(int nr_objs, int keep) /* * struct functions() - Map command to a function pointer. + * If you update this please update the documentation in slub_defrag.py */ struct functions { char *fn_name; diff --git a/tools/testing/slab/slub_defrag.py b/tools/testing/slab/slub_defrag.py new file mode 100755 index 000000000000..41747c0db39b --- /dev/null +++ b/tools/testing/slab/slub_defrag.py @@ -0,0 +1,451 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import subprocess +import sys +from os import path + +# SLUB Movable Objects test suite. +# +# Requirements: +# - CONFIG_SLUB=y +# - CONFIG_SLUB_DEBUG=y +# - The slub_defrag module in this directory. + +# Test SMO using a kernel module that enables triggering arbitrary +# kernel code from userspace via a debugfs file. +# +# Module code is in ./slub_defrag.c, basically the functionality is as +# follows: +# +# - Creates debugfs file /sys/kernel/debugfs/smo/callfn +# - Writes to 'callfn' are parsed as a command string and the function +# associated with command is called. +# - Defines 4 commands (all commands operate on smo_test cache): +# - 'test': Runs module stress tests. +# - 'alloc N': Allocates N slub objects +# - 'free N POS': Frees N objects starting at POS (see below) +# - 'enable': Enables SLUB Movable Objects +# +# The module maintains a list of allocated objects. Allocation adds +# objects to the tail of the list. Free'ing frees from the head of the +# list. This has the effect of creating free slots in the slab. For +# finer grained control over where in the cache slots are free'd POS +# (position) argument may be used. + +# The main() function is reasonably readable; the test suite does the +# following: +# +# 1. Runs the module stress tests. +# 2. Tests the cache without movable objects enabled. +# - Creates multiple partial slabs as explained above. +# - Verifies that partial slabs are _not_ removed by shrink (see below). +# 3. Tests the cache with movable objects enabled. +# - Creates multiple partial slabs as explained above. +# - Verifies that partial slabs _are_ removed by shrink (see below). + +# The sysfs file /sys/kernel/slab//shrink enables calling the +# function kmem_cache_shrink() (see mm/slab_common.c and mm/slub.cc). +# Shrinking a cache attempts to consolidate all partial slabs by moving +# objects if object migration is enable for the cache, otherwise +# shrinking a cache simply re-orders the partial list so as most densely +# populated slab are at the head of the list. + +# Enable/disable debugging output (also enabled via -d | --debug). +debug = False + +# Used in debug messages and when running `insmod`. +MODULE_NAME = "slub_defrag" + +# Slab cache created by the test module. +CACHE_NAME = "smo_test" + +# Set by get_slab_config() +objects_per_slab = 0 +pages_per_slab = 0 +debugfs_mounted = False # Set to true if we mount debugfs. + + +def eprint(*args, **kwargs): + print(*args, file=sys.stderr, **kwargs) + + +def dprint(*args, **kwargs): + if debug: + print(*args, file=sys.stderr, **kwargs) + + +def run_shell(cmd): + return subprocess.call([cmd], shell=True) + + +def run_shell_get_stdout(cmd): + return subprocess.check_output([cmd], shell=True) + + +def assert_root(): + user = run_shell_get_stdout('whoami') + if user != b'root\n': + eprint("Please run script as root") + sys.exit(1) + + +def mount_debugfs(): + mounted = False + + # Check if debugfs is mounted at a known mount point. + ret = run_shell('mount -l | grep /sys/kernel/debug > /dev/null 2>&1') + if ret != 0: + run_shell('mount -t debugfs none /sys/kernel/debug/') + mounted = True + dprint("Mounted debugfs on /sys/kernel/debug") + + return mounted + + +def umount_debugfs(): + dprint("Un-mounting debugfs") + run_shell('umount /sys/kernel/debug') + + +def load_module(): + """Loads the test module. + + We need a clean slab state to start with so module must + be loaded by the test suite. + """ + ret = run_shell('lsmod | grep %s > /dev/null' % MODULE_NAME) + if ret == 0: + eprint("Please unload slub_defrag module before running test suite") + return -1 + + dprint('Loading module ...') + ret = run_shell('insmod %s.ko' % MODULE_NAME) + if ret != 0: # ret==1 on error + return -1 + + dprint("Slab cache %s created" % CACHE_NAME) + return 0 + + +def unload_module(): + ret = run_shell('lsmod | grep %s > /dev/null' % MODULE_NAME) + if ret == 0: + dprint('Removing module %s ...' % MODULE_NAME) + run_shell('rmmod %s > /dev/null 2>&1' % MODULE_NAME) + + +def get_sysfs_value(filename): + """ + Parse slab sysfs files (single line: '20 N0=20') + """ + path = '/sys/kernel/slab/smo_test/%s' % filename + f = open(path, "r") + s = f.readline() + tokens = s.split(" ") + + return int(tokens[0]) + + +def get_nr_objects_active(): + return get_sysfs_value('objects') + + +def get_nr_objects_total(): + return get_sysfs_value('total_objects') + + +def get_nr_slabs_total(): + return get_sysfs_value('slabs') + + +def get_nr_slabs_partial(): + return get_sysfs_value('partial') + + +def get_nr_slabs_full(): + return get_nr_slabs_total() - get_nr_slabs_partial() + + +def get_slab_config(): + """Get relevant information from sysfs.""" + global objects_per_slab + + objects_per_slab = get_sysfs_value('objs_per_slab') + if objects_per_slab < 0: + return -1 + + dprint("Objects per slab: %d" % objects_per_slab) + return 0 + + +def verify_state(nr_objects_active, nr_objects_total, + nr_slabs_partial, nr_slabs_full, nr_slabs_total, msg=''): + err = 0 + got_nr_objects_active = get_nr_objects_active() + got_nr_objects_total = get_nr_objects_total() + got_nr_slabs_partial = get_nr_slabs_partial() + got_nr_slabs_full = get_nr_slabs_full() + got_nr_slabs_total = get_nr_slabs_total() + + if got_nr_objects_active != nr_objects_active: + err = -1 + + if got_nr_objects_total != nr_objects_total: + err = -2 + + if got_nr_slabs_partial != nr_slabs_partial: + err = -3 + + if got_nr_slabs_full != nr_slabs_full: + err = -4 + + if got_nr_slabs_total != nr_slabs_total: + err = -5 + + if err != 0: + dprint("Verify state: %s" % msg) + dprint(" what\t\t\twant\tgot") + dprint("-----------------------------------------") + dprint(" %s\t%d\t%d" % ('nr_objects_active', nr_objects_active, got_nr_objects_active)) + dprint(" %s\t%d\t%d" % ('nr_objects_total', nr_objects_total, got_nr_objects_total)) + dprint(" %s\t%d\t%d" % ('nr_slabs_partial', nr_slabs_partial, got_nr_slabs_partial)) + dprint(" %s\t\t%d\t%d" % ('nr_slabs_full', nr_slabs_full, got_nr_slabs_full)) + dprint(" %s\t%d\t%d\n" % ('nr_slabs_total', nr_slabs_total, got_nr_slabs_total)) + + return err + + +def exec_via_sysfs(command): + ret = run_shell('echo %s > /sys/kernel/debug/smo/callfn' % command) + if ret != 0: + eprint("Failed to echo command to sysfs: %s" % command) + + return ret + + +def enable_movable_objects(): + return exec_via_sysfs('enable') + + +def alloc(n): + exec_via_sysfs("alloc %d" % n) + + +def free(n, pos = 0): + exec_via_sysfs('free %d %d' % (n, pos)) + + +def shrink(): + ret = run_shell('slabinfo smo_test -s') + if ret != 0: + eprint("Failed to execute slabinfo -s") + + +def sanity_checks(): + # Verify everything is 0 to start with. + return verify_state(0, 0, 0, 0, 0, "sanity check") + + +def test_non_movable(): + one_over = objects_per_slab + 1 + + dprint("testing slab 'smo_test' prior to enabling movable objects ...") + + alloc(one_over) + + objects_active = one_over + objects_total = objects_per_slab * 2 + slabs_partial = 1 + slabs_full = 1 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: initial allocation") + if ret != 0: + eprint("test_non_movable: failed to verify initial state") + return -1 + + # Free object from first slot of first slab. + free(1) + objects_active = one_over - 1 + objects_total = objects_per_slab * 2 + slabs_partial = 2 + slabs_full = 0 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: after free") + if ret != 0: + eprint("test_non_movable: failed to verify after free") + return -1 + + # Non-movable cache, shrink should have no effect. + shrink() + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: after shrink") + if ret != 0: + eprint("test_non_movable: failed to verify after shrink") + return -1 + + # Cleanup + free(objects_per_slab) + shrink() + + dprint("verified non-movable slabs are NOT shrinkable") + return 0 + + +def test_movable(): + one_over = objects_per_slab + 1 + + dprint("testing slab 'smo_test' after enabling movable objects ...") + + alloc(one_over) + + objects_active = one_over + objects_total = objects_per_slab * 2 + slabs_partial = 1 + slabs_full = 1 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: initial allocation") + if ret != 0: + eprint("test_movable: failed to verify initial state") + return -1 + + # Free object from first slot of first slab. + free(1) + objects_active = one_over - 1 + objects_total = objects_per_slab * 2 + slabs_partial = 2 + slabs_full = 0 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: after free") + if ret != 0: + eprint("test_movable: failed to verify after free") + return -1 + + # movable cache, shrink should move objects and free slab. + shrink() + objects_active = one_over - 1 + objects_total = objects_per_slab * 1 + slabs_partial = 0 + slabs_full = 1 + slabs_total = 1 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: after shrink") + if ret != 0: + eprint("test_movable: failed to verify after shrink") + return -1 + + # Cleanup + free(objects_per_slab) + shrink() + + dprint("verified movable slabs are shrinkable") + return 0 + + +def dprint_start_test(test): + dprint("Running %s ..." % test) + + +def dprint_done(): + dprint("") + + +def run_test(fn, desc): + dprint_start_test(desc) + ret = fn() + if ret < 0: + fail_test(desc) + dprint_done() + + +# Load and unload the module for this test to ensure clean state. +def run_module_stress_test(): + dprint("Running module stress test (see dmesg for additional test output) ...") + + unload_module() + ret = load_module() + if ret < 0: + cleanup_and_exit(ret) + + exec_via_sysfs("test"); + + unload_module() + + dprint() + + +def fail_test(msg): + eprint("\nFAIL: test failed: '%s' ... aborting\n" % msg) + cleanup_and_exit(1) + + +def display_help(): + print("Usage: %s [OPTIONS]\n" % path.basename(sys.argv[0])) + print("\tRuns defrag test suite (a.k.a. SLUB Movable Objects)\n") + print("OPTIONS:") + print("\t-d | --debug Enable verbose debug output") + print("\t-h | --help Print this help and exit") + + +def cleanup_and_exit(return_code): + global debugfs_mounted + + if debugfs_mounted == True: + umount_debugfs() + + unload_module() + + sys.exit(return_code) + + +def main(): + global debug + + if len(sys.argv) > 1: + if sys.argv[1] == '-h' or sys.argv[1] == '--help': + display_help() + sys.exit(0) + + if sys.argv[1] == '-d' or sys.argv[1] == '--debug': + debug = True + + assert_root() + + # Use cleanup_and_exit() instead of sys.exit() after mounting debugfs. + debugfs_mounted = mount_debugfs() + + # Loads and unloads the module. + run_module_stress_test() + + ret = load_module() + if (ret < 0): + cleanup_and_exit(ret) + + ret = get_slab_config() + if (ret != 0): + fail_test("get slab config details") + + run_test(sanity_checks, "sanity checks") + + run_test(test_non_movable, "test non-movable") + + ret = enable_movable_objects() + if (ret != 0): + fail_test("enable movable objects") + + run_test(test_movable, "test movable") + + cleanup_and_exit(0) + +if __name__== "__main__": + main() From patchwork Wed Apr 3 04:21:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882757 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3170C17E1 for ; Wed, 3 Apr 2019 04:23:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A9AC287A6 for ; Wed, 3 Apr 2019 04:23:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0DF9D289BF; Wed, 3 Apr 2019 04:23:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7EA79287A6 for ; Wed, 3 Apr 2019 04:23:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 644CB6B026D; Wed, 3 Apr 2019 00:23:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5F5AF6B027A; Wed, 3 Apr 2019 00:23:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BDF56B027B; Wed, 3 Apr 2019 00:23:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 2EDBA6B026D for ; Wed, 3 Apr 2019 00:23:54 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id 23so13538792qkl.16 for ; Tue, 02 Apr 2019 21:23:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uAywuLhgM6+Szq4etbWPzwmP073FVonmktwU8/xEONg=; b=aMhMlIJGbF4QJ3XeSWpI7RPRl8AHO0cvNuM1KebXT12u66abNyLf4X+6f44L3RJKjY B6Y/iFYvvd2mpFvoGfCgHrmuSSOLye97Z4hB69a42Fyd3mOU0Ve3qpFqpCnQoTsiqruE +wCoEXCkYfKpgWOORAIVcHknaEtEb7UfrSsCkFPEO/iE1B9SVr96aC28lM7fCprVYSdS DUzaF+z6eIPP425snBsu0MwK0wBCjT9XyXjWpVL+AfqLvCj4a+yJzWHd9lMeAweOZkC3 9YXseuwVnueXXnq4KU1eOD2VMlhPVQA/ogAgeVM1PxDsZmjlAc+u8xAJB43dNmRqiUZx GAdA== X-Gm-Message-State: APjAAAVRTXkqs+lagCrEIGRhtBg1ii9P99OqisgeUsawlVoSFyByCL/n Qv0QGHLMwK29EtD5PeqeZQmyrZy4lRf5kfL2T3+2a6YNZmWWdnpznsfs8aid9VzHFHsgpLDyvbi ApAzMGQFoZEqAeXxW61F3fuFFIqw7L8o+dxln97JAlohx4MT2FDo85QDED+L+YSM= X-Received: by 2002:a05:620a:132b:: with SMTP id p11mr93947qkj.279.1554265433938; Tue, 02 Apr 2019 21:23:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqzQn4UMtrPQ0Om/0T/QiDecaolJ1rWFJ39GPw76cJ4OEF0UfbvVKjZv7BJSiyFxkTPSSeV3 X-Received: by 2002:a05:620a:132b:: with SMTP id p11mr93899qkj.279.1554265432875; Tue, 02 Apr 2019 21:23:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265432; cv=none; d=google.com; s=arc-20160816; b=uIkEHvvb4mmgTiQnSPBDFSBqOUAk3WuhhbX10BGkZa8dbK/7Mcui9/rjJrlh8Orx2v FPAtHOYLi/QBanr0TciMFGGWQaS+q/UxCnvXsq/Czv6FQS6qhF00IYMAdU3M/50OiEWX IO15I67pmI9gOPCbiRqV2Yq1Hby49h0SABmFhAHqg7emUWCkp8h93i1Kt/pCX+Ax+EHI MSEXUpls+R74yesvHJPmBYyA37tIY7TiLuoC0ifM/p80wDzo+WdamLulaJZh8U4GEAqg VC3mMYq/2nMmupyHwI8xnVWU2eSjjeYaRYVLJES3lXwU8DYv/RP8tuHq2vE7omV6lSi6 bmKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=uAywuLhgM6+Szq4etbWPzwmP073FVonmktwU8/xEONg=; b=F8cvRrBapqE55PJ9lcscjaJF7QzMvEHINwvQRYsBzO84LG9HueZ+tAFohxOWtuEybh YB96EPNzb7LkppAN1eX3Tsqk2/lbrCHu1gu1AOl66wJSxyoQKL5CRRHCSILnAXSo+yiB risdMt0f6AB9Zgapy3+Py/onXopMPxxB92CFoO2CdZAU6vY7/cQrcqgrdpe7PYWGrNyB RahXJEthxWDp7zgAciP+5sDxog6zO2e68uOvsfMNqYYv/9uh7zEp+AtN3OUtLJu6Q3Ax AE/g4G9n13ltN9b/aZjG97tn/+KdhxAMrLztDs3VgCU1AzAeEkXqoH0UuoWPG5MlavXz 8NzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=xeXsXdNf; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id n64si2500264qte.106.2019.04.02.21.23.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:23:52 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=xeXsXdNf; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 91FB421B84; Wed, 3 Apr 2019 00:23:52 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:23:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=uAywuLhgM6+Szq4etbWPzwmP073FVonmktwU8/xEONg=; b=xeXsXdNf P0XevguDVWe8R0wWT3qYLmuo5crZKYPd4pltu4WJxBU0f1oEsa362LSpgZ4CdNQZ InDlUyX3uK85y8/EYAAhVz5TMUtIHMgU5XXYa5LIjJPFILKnrfjMeJrdXL/dSU6Y pJlOZ3Mo836ihiKeOtTmLe+/CGXJQ8tmDK3FsS8LQUztBoDAC5+NA+JnSY/Zs4F4 o+IXwNfDR69U8p86UcIwY8UpCknhFqteRa5q7IUTo/MbwI//s1f3+TKLxZTEVbA8 HKFIpylGzC69vJxjlgNyf1x9aio4JulIFvN5CtZCFtfVq2Sk50jrULAcxMsV3Mx1 hz3eqk7G3QXNhg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeek X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 9597A100E5; Wed, 3 Apr 2019 00:23:45 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 09/14] xarray: Implement migration function for objects Date: Wed, 3 Apr 2019 15:21:22 +1100 Message-Id: <20190403042127.18755-10-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Implement functions to migrate objects. This is based on initial code by Matthew Wilcox and was modified to work with slab object migration. Cc: Matthew Wilcox Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- lib/radix-tree.c | 13 +++++++++++++ lib/xarray.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 14d51548bea6..9412c2853726 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -1613,6 +1613,17 @@ static int radix_tree_cpu_dead(unsigned int cpu) return 0; } +extern void xa_object_migrate(void *tree_node, int numa_node); + +static void radix_tree_migrate(struct kmem_cache *s, void **objects, int nr, + int node, void *private) +{ + int i; + + for (i = 0; i < nr; i++) + xa_object_migrate(objects[i], node); +} + void __init radix_tree_init(void) { int ret; @@ -1627,4 +1638,6 @@ void __init radix_tree_init(void) ret = cpuhp_setup_state_nocalls(CPUHP_RADIX_DEAD, "lib/radix:dead", NULL, radix_tree_cpu_dead); WARN_ON(ret < 0); + kmem_cache_setup_mobility(radix_tree_node_cachep, NULL, + radix_tree_migrate); } diff --git a/lib/xarray.c b/lib/xarray.c index 6be3acbb861f..6d2657f2e4cb 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1971,6 +1971,52 @@ void xa_destroy(struct xarray *xa) } EXPORT_SYMBOL(xa_destroy); +void xa_object_migrate(struct xa_node *node, int numa_node) +{ + struct xarray *xa = READ_ONCE(node->array); + void __rcu **slot; + struct xa_node *new_node; + int i; + + /* Freed or not yet in tree then skip */ + if (!xa || xa == XA_RCU_FREE) + return; + + new_node = kmem_cache_alloc_node(radix_tree_node_cachep, + GFP_KERNEL, numa_node); + if (!new_node) + return; + + xa_lock_irq(xa); + + /* Check again..... */ + if (xa != node->array || !list_empty(&node->private_list)) { + node = new_node; + goto unlock; + } + + memcpy(new_node, node, sizeof(struct xa_node)); + + /* Move pointers to new node */ + INIT_LIST_HEAD(&new_node->private_list); + for (i = 0; i < XA_CHUNK_SIZE; i++) { + void *x = xa_entry_locked(xa, new_node, i); + + if (xa_is_node(x)) + rcu_assign_pointer(xa_to_node(x)->parent, new_node); + } + if (!new_node->parent) + slot = &xa->xa_head; + else + slot = &xa_parent_locked(xa, new_node)->slots[new_node->offset]; + rcu_assign_pointer(*slot, xa_mk_node(new_node)); + +unlock: + xa_unlock_irq(xa); + xa_node_free(node); + rcu_barrier(); +} + #ifdef XA_DEBUG void xa_dump_node(const struct xa_node *node) { From patchwork Wed Apr 3 04:21:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882761 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9B1D31575 for ; Wed, 3 Apr 2019 04:24:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 835BE287A6 for ; Wed, 3 Apr 2019 04:24:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 768AF289BF; Wed, 3 Apr 2019 04:24:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 88414287A6 for ; Wed, 3 Apr 2019 04:24:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 840FD6B027B; Wed, 3 Apr 2019 00:24:02 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7F1C46B027C; Wed, 3 Apr 2019 00:24:02 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B97F6B027D; Wed, 3 Apr 2019 00:24:02 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 4AC896B027B for ; Wed, 3 Apr 2019 00:24:02 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id 77so13640962qkd.9 for ; Tue, 02 Apr 2019 21:24:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=d2hJTA3JxRvBa5Hf3iT2erlIYVlWyqv0FDpOPG34bXsHDwUvxLplPuaYRSNMvjQ+5F AfttdnZnbLLRzt+9kGSqRAuMhy0VCQFzI62LeHX5F1NX12La7YsyaBDPNM2dS9NJoTFy HNP/UDHS2v2zjX7EjDfGiBwblmbxWoNSEt7/+tbUJra9siHLBtR8jBVXMFRFCtP/Anqv Nh2OSWCyNgXOLdzVX+750Qy2+o0nQ1uTjqLi5KI3kMoL0t61cP9QPTVHIVc1DbQNmfkn 5VigGQN8uTcfF6tPbJX89ZnIchf8dpzN8WAB2oV63CBV6iMm/GAhM3Hq0jHtrnrru2m+ Ra4w== X-Gm-Message-State: APjAAAXYrLYp8xfbI73wwsrN9XRx7nWpI6SAW6CS92ywn6YcR5gicqyV 4WHyme/VSgd0ihwrDz/+7E7i7S8m2WaiMCObfsJxm09iAMPL0pLW8D0nasIOYj9A9Pn5fcR0VYL /LjvIByDPUxKXvZXuQ1jtsZEnpnBCoWzfQZD8ANdre0B8g44HdQcRrSxDZbHvv14= X-Received: by 2002:ac8:2c5a:: with SMTP id e26mr34373263qta.205.1554265442039; Tue, 02 Apr 2019 21:24:02 -0700 (PDT) X-Google-Smtp-Source: APXvYqw1Dqu+TxBucd3ph+8aC/UuZG8pMNH6DT6iEv86d0jJ2UKJ9m4pvP380TeD5dv58UlySzJp X-Received: by 2002:ac8:2c5a:: with SMTP id e26mr34373212qta.205.1554265440558; Tue, 02 Apr 2019 21:24:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265440; cv=none; d=google.com; s=arc-20160816; b=ZKjK9BDtbjLQrX6QaMclU22COotw1RhLUlkZpUYWcw1LLGSZL14v668B21gZsr9wzo 78xpinozwQvHqBbIVS/3rGeDrYiWVlMK4D17lq5UCENJo2DIfAQHiz2AlP6Gv0Wc8P9e q5c5TFatQarFspECfnYo5iUAeQvKMvlqmRmLvJ6eoHaW1UcbycVvaZ8Y1ZVchnxzHLfk UzKEd5qNXiYhNBxRGRL8qxsz7Qyds3/tWTJWTe95YscJ0sixRVg2TMxsnBcBFKJy62xY kwQe9Ra6X3EmbWKBmO6i1JWlV5weQhfh5SkOShRG6as+SXc2hjoSEVy7l4RbdQTfXOdw zu6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=gtQhbUgL/H8lKWW0FSlmtwGXnl6j5FsmrwsnRS+SyINkxalyxBNhWEj4tY/2O0LtwQ lyND6IJhzB2IYNiWh93OUBfyV3i3nb6DbKgTnapGBRqKgTq2e/tWq+Xw4ZobTVstAr3y n/205JPTWobm5+I7X8bI5aKgeAER9VRRbO2No8fAjoUQ/ffnzzT4x9LvH4skuH24nXJ3 KXYFEPoiL4QgSKvG0lPGqiwMinaqrwzGyocOsAuWpEXFtzJA0ZDa5hk/hMp9ngQu5Dtb ohF0ixpDYnzR4avsSoqF600uxg54Ox1ms0buNkKRKicxdgVQ8fibKtRfgixb6FkkSXat KxmA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ZLjFmVPc; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id a24si1567957qkk.138.2019.04.02.21.24.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:24:00 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ZLjFmVPc; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 2063D21E65; Wed, 3 Apr 2019 00:24:00 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:24:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=ZLjFmVPc uFn07xt2EXbinYN6llUG0QEjaIi3wr2dZiGZF4j8TwtA4AshpAEgA7xPhxbPcHpB x83ISJhNskBGJ0YiWUMUnw/aSYAKXvgWvbMv24+SHHtaPQfyz9nInO2rwVmCa+xH VA2HjXG+GxmdwAcecLJyYXlvHPQFFX3AjpXJ+73aIv+9AcCHwZ1xG9MNQXSqczuv 8M0WomHfwYqG+HWQni461sG7HneqZuOZjozJlvB6IrmI/btF2Gzj/PRVgKUiqVwa Y5+1koLJgjVbP7fPD54FEbziXd96+emuuJsefJJJqlRJCRj1jQg7jyheNzTeaSXT mSMK6evUnkOfow== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeel X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 214AB100E5; Wed, 3 Apr 2019 00:23:52 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 10/14] tools/testing/slab: Add XArray movable objects tests Date: Wed, 3 Apr 2019 15:21:23 +1100 Message-Id: <20190403042127.18755-11-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just implemented movable objects for the XArray. Let's test it intree. Add test module for the XArray's movable objects implementation. Functionality of the XArray Slab Movable Object implementation can usually be seen by simply by using `slabinfo` on a running machine since the radix tree is typically in use on a running machine and will have partial slabs. For repeated testing we can use the test module to run to simulate a workload on the XArray then use `slabinfo` to test object migration is functioning. If testing on freshly spun up VM (low radix tree workload) it may be necessary to load/unload the module a number of times to create partial slabs. Example test session -------------------- Relevant /proc/slabinfo column headers: name Prior to testing slabinfo report for radix_tree_node: # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8352 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 497 Sanity Checks : On Total: 8142848 SlabObj: 912 Full : 473 Redzoning : On Used : 4810752 SlabSiz: 16384 Partial: 24 Poisoning : On Loss : 3332096 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2806272 Align : 8 Objects: 17 Tracing : Off Lpadd: 437360 Here you can see the kernel was built with Slab Movable Objects enabled for the XArray (XArray uses the radix tree below the surface). After inserting the test module (note we have triggered allocation of a number of radix tree nodes increasing the object count but decreasing the number of partial slabs): # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8442 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 499 Sanity Checks : On Total: 8175616 SlabObj: 912 Full : 484 Redzoning : On Used : 4862592 SlabSiz: 16384 Partial: 15 Poisoning : On Loss : 3313024 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2836512 Align : 8 Objects: 17 Tracing : Off Lpadd: 439120 Now we can shrink the radix_tree_node cache: # slabinfo radix_tree_node --shrink # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8515 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 501 Sanity Checks : On Total: 8208384 SlabObj: 912 Full : 500 Redzoning : On Used : 4904640 SlabSiz: 16384 Partial: 1 Poisoning : On Loss : 3303744 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2861040 Align : 8 Objects: 17 Tracing : Off Lpadd: 440880 Note the single remaining partial slab. Signed-off-by: Tobin C. Harding --- tools/testing/slab/Makefile | 2 +- tools/testing/slab/slub_defrag_xarray.c | 211 ++++++++++++++++++++++++ 2 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 tools/testing/slab/slub_defrag_xarray.c diff --git a/tools/testing/slab/Makefile b/tools/testing/slab/Makefile index 440c2e3e356f..44c18d9a4d52 100644 --- a/tools/testing/slab/Makefile +++ b/tools/testing/slab/Makefile @@ -1,4 +1,4 @@ -obj-m += slub_defrag.o +obj-m += slub_defrag.o slub_defrag_xarray.o KTREE=../../.. diff --git a/tools/testing/slab/slub_defrag_xarray.c b/tools/testing/slab/slub_defrag_xarray.c new file mode 100644 index 000000000000..41143f73256c --- /dev/null +++ b/tools/testing/slab/slub_defrag_xarray.c @@ -0,0 +1,211 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SMOX_CACHE_NAME "smox_test" +static struct kmem_cache *cachep; + +/* + * Declare XArrays globally so we can clean them up on module unload. + */ + +/* Used by test_smo_xarray()*/ +DEFINE_XARRAY(things); + +/* Thing to store pointers to in the XArray */ +struct smox_thing { + long id; +}; + +/* It's up to the caller to ensure id is unique */ +static struct smox_thing *alloc_thing(int id) +{ + struct smox_thing *thing; + + thing = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!thing) + return ERR_PTR(-ENOMEM); + + thing->id = id; + return thing; +} + +/** + * smox_object_ctor() - SMO object constructor function. + * @ptr: Pointer to memory where the object should be constructed. + */ +void smox_object_ctor(void *ptr) +{ + struct smox_thing *thing = ptr; + + thing->id = -1; +} + +/** + * smox_cache_migrate() - kmem_cache migrate function. + * @cp: kmem_cache pointer. + * @objs: Array of pointers to objects to migrate. + * @size: Number of objects in @objs. + * @node: NUMA node where the object should be allocated. + * @private: Pointer returned by kmem_cache_isolate_func(). + */ +void smox_cache_migrate(struct kmem_cache *cp, void **objs, int size, + int node, void *private) +{ + struct smox_thing **ptrs = (struct smox_thing **)objs; + struct smox_thing *old, *new; + struct smox_thing *thing; + unsigned long index; + void *entry; + int i; + + for (i = 0; i < size; i++) { + old = ptrs[i]; + + new = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!new) { + pr_debug("kmem_cache_alloc failed\n"); + return; + } + + new->id = old->id; + + /* Update reference the brain dead way */ + xa_for_each(&things, index, thing) { + if (thing == old) { + entry = xa_store(&things, index, new, GFP_KERNEL); + if (entry != old) { + pr_err("failed to exchange new/old\n"); + return; + } + } + } + kmem_cache_free(cachep, old); + } +} + +/* + * test_smo_xarray() - Run some tests using an XArray. + */ +static int test_smo_xarray(void) +{ + const int keep = 6; /* Free 5 out of 6 items */ + const int nr_items = 10000; + struct smox_thing *thing; + unsigned long index; + void *entry; + int expected; + int i; + + /* + * Populate XArray, this adds to the radix_tree_node cache as + * well as the smox_test cache. + */ + for (i = 0; i < nr_items; i++) { + thing = alloc_thing(i); + entry = xa_store(&things, i, thing, GFP_KERNEL); + if (xa_is_err(entry)) { + pr_err("smox: failed to allocate entry: %d\n", i); + return -ENOMEM; + } + } + + /* Now free items, putting holes in both caches. */ + for (i = 0; i < nr_items; i++) { + if (i % keep == 0) + continue; + + thing = xa_erase(&things, i); + if (xa_is_err(thing)) + pr_err("smox: error erasing entry: %d\n", i); + kmem_cache_free(cachep, thing); + } + + expected = 0; + xa_for_each(&things, index, thing) { + if (thing->id != expected || index != expected) { + pr_err("smox: error; got %ld want %d at %ld\n", + thing->id, expected, index); + return -1; + } + expected += keep; + } + + /* + * Leave caches sparsely allocated. Shrink caches manually with: + * + * slabinfo radix_tree_node --shrink + * slabinfo smox_test --shrink + */ + + return 0; +} + +static int __init smox_cache_init(void) +{ + cachep = kmem_cache_create(SMOX_CACHE_NAME, + sizeof(struct smox_thing), + 0, 0, smox_object_ctor); + if (!cachep) + return -1; + + return 0; +} + +static void __exit smox_cache_cleanup(void) +{ + struct smox_thing *thing; + unsigned long i; + + xa_for_each(&things, i, thing) { + kmem_cache_free(cachep, thing); + } + xa_destroy(&things); + kmem_cache_destroy(cachep); +} + +static int __init smox_init(void) +{ + int ret; + + ret = smox_cache_init(); + if (ret) { + pr_err("smo_xarray: failed to create cache\n"); + return ret; + } + pr_info("smo_xarray: created kmem_cache: %s\n", SMOX_CACHE_NAME); + + kmem_cache_setup_mobility(cachep, NULL, smox_cache_migrate); + pr_info("smo_xarray: kmem_cache %s defrag enabled\n", SMOX_CACHE_NAME); + + /* + * Running this test consumes memory unless you shrink the + * radix_tree_node cache manually with `slabinfo`. + */ + ret = test_smo_xarray(); + if (ret) + pr_warn("test_smo_xarray failed: %d\n", ret); + + pr_info("smo_xarray: module loaded successfully\n"); + return 0; +} +module_init(smox_init); + +static void __exit smox_exit(void) +{ + smox_cache_cleanup(); + + pr_info("smo_xarray: module removed\n"); +} +module_exit(smox_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Tobin C. Harding"); +MODULE_DESCRIPTION("SMO XArray test module."); From patchwork Wed Apr 3 04:21:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882759 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 337041575 for ; Wed, 3 Apr 2019 04:24:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16F4C289B5 for ; Wed, 3 Apr 2019 04:24:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 08401289C7; Wed, 3 Apr 2019 04:24:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 29AC5289B5 for ; Wed, 3 Apr 2019 04:24:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1153E6B0272; Wed, 3 Apr 2019 00:24:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0C6786B027C; Wed, 3 Apr 2019 00:24:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ED0936B027D; Wed, 3 Apr 2019 00:24:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id CB73F6B0272 for ; Wed, 3 Apr 2019 00:24:09 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id d8so13534016qkk.17 for ; Tue, 02 Apr 2019 21:24:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=YTpDBiPIz+r0uG1auSa8ph45UtBEpiDJlB6LNiHXZmQ=; b=cBzVLJ2+Fqsq28lj4PMoYCkuAt6m3O9pYQ0XQxLVKRpy9DP9WzqpMa8+P+40gH7Yqi wmR+MTemPVZ/PLfL/VW4cHaX/VwcGqse2C14clZRfCrBEqWSjzAgss2CvY6esZIMgJpO 1FU3diMea/gKrtvtGWMLB0a0ygbZKcEMT27iFa4JbykkL3GExACgPwPFv3egOYboKOsp 07j2uMkMj0qWPaDweOCr22E62kZdPNplrtDhrzvhQkji7shzTnNIEz90ybU+O3Xpvfxw I1bM8glgfkbYTBshfq8gMqkPp2JQE8Igf7tw5Fp0jeS5lzPe2kl6tMiYBtimYJFR9/5+ uFAw== X-Gm-Message-State: APjAAAWajbNNgtrXX61hSj6jdBkLLaMZFi1wbFQVlfiSNnvy06uDFUn4 +ADNL93bNAMNToHnLW/jO0FSiv2jMz7ezMjX5xbPjtTBfvFynB+UIcwlwt62F66ckvaRjI54fnC NEJN4S/YKgyCXXm+CJkvYwgAnP10CTnAsfXxmFFA7+dVB3mslYH1nAaSmcX8pxZ4= X-Received: by 2002:a0c:ba8b:: with SMTP id x11mr27193qvf.196.1554265449489; Tue, 02 Apr 2019 21:24:09 -0700 (PDT) X-Google-Smtp-Source: APXvYqykBRLl2pId63Z7GP2LLw5sLBBLVnd/a9QQYnjQONkGA4wUiz/KXxBvN6PZFR4h2pqOK3ZD X-Received: by 2002:a0c:ba8b:: with SMTP id x11mr27140qvf.196.1554265447915; Tue, 02 Apr 2019 21:24:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265447; cv=none; d=google.com; s=arc-20160816; b=JBMi0yYyv3DA5b7GimfwH4SpeTdYj+8Z21yVZj74f0Zds8DAoEYpaU3kGt0nuQRbIe igfciJ/u7ck6tWtiPKJa1s+sTlu3izY3Mzq2yIHat0WjGM8zyon54jUkbFjR6V64o9GM 6XYNXHsIL9OrChMq5bZRL6QKWjPupXCZlEVQizEAz8K8+m0B9jlNnN9LA9fC970Q8W8l eKTSBeh2WNuiTGORSc/bZu/0rkweiQxLN+wZI5FCsY5wJ5NkIglMU0Cd069DrR1j1iwb pimDhxVspqpY/QrJwVs/BQqAvPsVhFspkXP7bICfMW8W+g8KDnsSu0Ko1DlL6K3X+Xpu 409A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=YTpDBiPIz+r0uG1auSa8ph45UtBEpiDJlB6LNiHXZmQ=; b=z0RfU145R37zcPvNFWjF0tDxrTTz9rp76aD+cqCPPW8ds4RvCZw2rvWA3j4q6dZDnN 8aG/jF3WR/z+Wdozt+potctBj0BLa+sYVBiu9ZLCzM9mUqtfIAVw2z5giOqgwmaxhK0m CeVM78o9AVhCQAxM7IO2qSEdA+Gatmny4JIkmrgw5Qh8AaRLx6uJj2iAd7hZauTZt9tR Px3Um4Waa9I/QVjdVebhoOGaIHt4qo9gc9St/enhPFnBJtztQYRwkI78w2ojai+lEfTu uPBAkgdcKhwrSrqlDH0wnvVG0wVi427w1qxKSEaMyJvN+rHFtpG4bEQh5l1JKWt+A3F1 hzRA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=X5wsJ1NW; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id q31si2853875qtq.129.2019.04.02.21.24.07 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:24:07 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=X5wsJ1NW; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 9A09121F58; Wed, 3 Apr 2019 00:24:07 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:24:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=YTpDBiPIz+r0uG1auSa8ph45UtBEpiDJlB6LNiHXZmQ=; b=X5wsJ1NW j0kMUfLEpoua6IDApdgX2D7yPdMhSZdgZn7LQCeXgAfFApbVJApH41rmM3CuhuKQ kIlIUBmlYtXnf0ndhfiei7+kKgVJHfbvsQDmXo2IOGqS6mTQvjPc2gXqlMcwa3KJ BlBZy2FIJ5WYCsvUHiYkrX1e54/P3NaF3ZwhwTynoqeyZFshk7Gh/F55Ts3i2H+/ SfWKOzPAI46ATOYN4MoZ3vo/GUoQ3YZwjV86VWv5gNU/8amiMZQh+/12BXgN7oee LSE4FUYnRz9aOZg4uCa+QIuFODjrChMraumCeGxQFxct5Q5xN1A3SpAglREARmcB KN8fNlHrXPRUBw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpedutd X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 909C9100E5; Wed, 3 Apr 2019 00:24:00 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 11/14] slub: Enable moving objects to/from specific nodes Date: Wed, 3 Apr 2019 15:21:24 +1100 Message-Id: <20190403042127.18755-12-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We have just implemented Slab Movable Objects (object migration). Currently object migration is used to defrag a cache. On NUMA systems it would be nice to be able to control the source and destination nodes when moving objects. Add CONFIG_SMO_NODE to guard this feature. CONFIG_SMO_NODE depends on CONFIG_SLUB_DEBUG because we use the full list. Leave it like this for the RFC because the patch will be less cluttered to review, separate full list out of CONFIG_DEBUG before doing a PATCH version. Implement moving all objects (including those in full slabs) to a specific node. Expose this functionality to userspace via a sysfs entry. Add sysfs entry: /sysfs/kernel/slab//move With this users get access to the following functionality: - Move all objects to specified node. echo "N1" > move - Move all objects from specified node to other specified node (from N1 -> to N2): echo "N1 N2" > move This also enables shrinking slabs on a specific node: echo "N1 N1" > move Signed-off-by: Tobin C. Harding --- mm/Kconfig | 7 ++ mm/slub.c | 249 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 256 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 25c71eb8a7db..47040d939f3b 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -258,6 +258,13 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION config ARCH_ENABLE_THP_MIGRATION bool +config SMO_NODE + bool "Enable per node control of Slab Movable Objects" + depends on SLUB && SYSFS + select SLUB_DEBUG + help + On NUMA systems enable moving objects to and from a specified node. + config PHYS_ADDR_T_64BIT def_bool 64BIT diff --git a/mm/slub.c b/mm/slub.c index e601c804ed79..e4f3dde443f5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4345,6 +4345,106 @@ static void move_slab_page(struct page *page, void *scratch, int node) s->migrate(s, vector, count, node, private); } +#ifdef CONFIG_SMO_NODE +/* + * kmem_cache_move() - Attempt to move all slab objects. + * @s: The cache we are working on. + * @node: The node to move objects away from. + * @target_node: The node to move objects on to. + * + * Attempts to move all objects (partial slabs and full slabs) to target + * node. + * + * Context: Takes the list_lock. + * Return: The number of slabs remaining on node. + */ +static unsigned long kmem_cache_move(struct kmem_cache *s, + int node, int target_node) +{ + struct kmem_cache_node *n = get_node(s, node); + LIST_HEAD(move_list); + struct page *page, *page2; + unsigned long flags; + void **scratch; + + if (!s->migrate) { + pr_warn("%s SMO not enabled, cannot move objects\n", s->name); + goto out; + } + + scratch = alloc_scratch(s); + if (!scratch) + goto out; + + spin_lock_irqsave(&n->list_lock, flags); + + list_for_each_entry_safe(page, page2, &n->partial, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + if (page->inuse) { + list_move(&page->lru, &move_list); + /* Stop page being considered for allocations */ + n->nr_partial--; + page->frozen = 1; + + slab_unlock(page); + } else { /* Empty slab page */ + list_del(&page->lru); + n->nr_partial--; + slab_unlock(page); + discard_slab(s, page); + } + } + list_for_each_entry_safe(page, page2, &n->full, lru) { + if (!slab_trylock(page)) + continue; + + list_move(&page->lru, &move_list); + page->frozen = 1; + slab_unlock(page); + } + + spin_unlock_irqrestore(&n->list_lock, flags); + + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + + /* Bail here to save taking the list_lock */ + if (list_empty(&move_list)) + goto out; + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + if (page->inuse == page->objects) { + list_add(&page->lru, &n->full); + slab_unlock(page); + } else { + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); +out: + return atomic_long_read(&n->nr_slabs); +} +#endif /* CONFIG_SMO_NODE */ + /* * kmem_cache_defrag() - Defragment node. * @s: cache we are working on. @@ -4459,6 +4559,32 @@ static unsigned long kmem_cache_defrag(struct kmem_cache *s, return n->nr_partial; } +#ifdef CONFIG_SMO_NODE +/* + * kmem_cache_move_to_node() - Move all slab objects to node. + * @s: The cache we are working on. + * @node: The target node to move objects to. + * + * Attempt to move all slab objects from all nodes to @node. + * + * Return: The total number of slabs left on emptied nodes. + */ +static unsigned long kmem_cache_move_to_node(struct kmem_cache *s, int node) +{ + unsigned long left = 0; + int nid; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == node) + continue; + + left += kmem_cache_move(s, nid, node); + } + + return left; +} +#endif + /** * kmem_defrag_slabs() - Defrag slab caches. * @node: The node to defrag or -1 for all nodes. @@ -5603,6 +5729,126 @@ static ssize_t shrink_store(struct kmem_cache *s, } SLAB_ATTR(shrink); +#ifdef CONFIG_SMO_NODE +static ssize_t move_show(struct kmem_cache *s, char *buf) +{ + return 0; +} + +/* + * parse_move_store_input() - Parse buf getting integer arguments. + * @buf: Buffer to parse. + * @length: Length of @buf. + * @arg0: Return parameter, first argument. + * @arg1: Return parameter, second argument. + * + * Parses the input from user write to sysfs file 'move'. Input string + * should contain either one or two node specifiers of form Nx where x + * is an integer specifying the NUMA node ID. 'N' or 'n' may be used. + * n/N may be omitted. + * + * e.g. + * echo 'N1' > /sysfs/kernel/slab/cache/move + * or + * echo 'N0 N2' > /sysfs/kernel/slab/cache/move + * + * Regex matching accepted forms: '[nN]?[0-9]( [nN]?[0-9])?' + * + * FIXME: This is really fragile. Input must be exactly correct, + * spurious whitespace causes parse errors. + * + * Return: 0 if an argument was successfully converted, or an error code. + */ +static ssize_t parse_move_store_input(const char *buf, size_t length, + long *arg0, long *arg1) +{ + char *s, *save, *ptr; + int ret = 0; + + if (!buf) + return -EINVAL; + + s = kstrdup(buf, GFP_KERNEL); + if (!s) + return -ENOMEM; + save = s; + + if (s[length - 1] == '\n') { + s[length - 1] = '\0'; + length--; + } + + ptr = strsep(&s, " "); + if (!ptr || strcmp(ptr, "") == 0) { + ret = 0; + goto out; + } + + if (*ptr == 'N' || *ptr == 'n') + ptr++; + ret = kstrtol(ptr, 10, arg0); + if (ret < 0) + goto out; + + if (s) { + if (*s == 'N' || *s == 'n') + s++; + ret = kstrtol(s, 10, arg1); + if (ret < 0) + goto out; + } + + ret = 0; +out: + kfree(save); + return ret; +} + +static bool is_valid_node(int node) +{ + int nid; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == node) + return true; + } + return false; +} + +/* + * move_store() - Move objects between nodes. + * @s: The cache we are working on. + * @buf: String received. + * @length: Length of @buf. + * + * Writes to /sys/kernel/slab//move are interpreted as follows: + * + * echo "N1" > move : Move all objects (from all nodes) to node 1. + * echo "N0 N1" > move : Move all objects from node 0 to node 1. + * + * 'N' may be omitted: + */ +static ssize_t move_store(struct kmem_cache *s, const char *buf, size_t length) +{ + long arg0 = -1; + long arg1 = -1; + int ret; + + ret = parse_move_store_input(buf, length, &arg0, &arg1); + if (ret < 0) + return -EINVAL; + + if (is_valid_node(arg0) && is_valid_node(arg1)) + (void)kmem_cache_move(s, arg0, arg1); + else if (is_valid_node(arg0)) + (void)kmem_cache_move_to_node(s, arg0); + + /* FIXME: What should we be returning here? */ + return length; +} +SLAB_ATTR(move); +#endif /* CONFIG_SMO_NODE */ + #ifdef CONFIG_NUMA static ssize_t remote_node_defrag_ratio_show(struct kmem_cache *s, char *buf) { @@ -5727,6 +5973,9 @@ static struct attribute *slab_attrs[] = { &reclaim_account_attr.attr, &destroy_by_rcu_attr.attr, &shrink_attr.attr, +#ifdef CONFIG_SMO_NODE + &move_attr.attr, +#endif &slabs_cpu_partial_attr.attr, #ifdef CONFIG_SLUB_DEBUG &total_objects_attr.attr, From patchwork Wed Apr 3 04:21:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882763 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CC60C17E1 for ; Wed, 3 Apr 2019 04:24:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B3365287A6 for ; Wed, 3 Apr 2019 04:24:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A6269289BF; Wed, 3 Apr 2019 04:24:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF2F6287A6 for ; Wed, 3 Apr 2019 04:24:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B9C546B027C; Wed, 3 Apr 2019 00:24:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B4B816B027D; Wed, 3 Apr 2019 00:24:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A15216B027E; Wed, 3 Apr 2019 00:24:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 7E45F6B027C for ; Wed, 3 Apr 2019 00:24:16 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id e31so11510847qtb.0 for ; Tue, 02 Apr 2019 21:24:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=aa6iExGIwghZDn1ZohvQXbuCEdzySOldf4bort+1eX4=; b=Ey0HEKq65UA29PpjVvRlXWV30nw9MY3iwHs5dzXTrx5YJOuaZRx69lf07GCnJbKH1k eSN8kH84r/CDFHXkSs+hOnZXbClLiW8ZjeQuTO+B307trd8c27mRyxLW2N7Z6R0Jyo+B FeVhOMQNC1jcwHA0N6cogXcGmNjADhHguF6an+DXmdzT0961JA8ph/EcRY8WjxZWDIda FkE45ffCZ+KPbL0re/c8aODvktz8+mj/EJoUEQ4qzXx5J60ThRhyZSCfUrwTY2iCoBAe 7lshE+ZnM2zBmlEUsaHiwzpk+ymLqQK0ANHsRuO2OqAjoHKc0ai6UcCc9tXiZIlRelzz xZxw== X-Gm-Message-State: APjAAAXmnI8GRF6sZLUWbVkj6UdyTdTybnEaXTOnhw8pVtlFji0sf7zr jmmNFc0gMoDwzMagcdH+W+xYBSGlEEkBziiZA1dHL5RGTRr/e2pb0iSVlPlXQ67NBDEUWO2wnoi JwxFujXyj6smA0cjhZdeNfMuEDvW4OHGtKcxPoDE1Li9HFEmkJKlgG5ukW99hERU= X-Received: by 2002:aed:2bc7:: with SMTP id e65mr52285932qtd.339.1554265456257; Tue, 02 Apr 2019 21:24:16 -0700 (PDT) X-Google-Smtp-Source: APXvYqwWagwU+/Z5sBioysnZZcQ6YVp+xuenla8Pw0qvWj75306GOkS5+sgUsPeFBw+mv42A7ds7 X-Received: by 2002:aed:2bc7:: with SMTP id e65mr52285890qtd.339.1554265455286; Tue, 02 Apr 2019 21:24:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265455; cv=none; d=google.com; s=arc-20160816; b=Tfm+rM+1Ts2nmCu31z8hFyUOdzP6iaXb01KnYsc3R8D8QSjkE4XCYRony8sFKfHvzF 6dfTdUmqDxf5EwQ3Wt3Wk446hESDoNyXVpNC+bAnClPZsdNXAHxGZZblYmu6FUoJ6UeL 79AGoMyYi+Z0egulYaYh3hAVDDae7z8zeshfQvyVyTrvD8QiaXmup45CMfl1Ktz6xx3J BoI+wrocaZiE8rDwL6noLMPQ774/4DzU40SHcdxkU+8418mQ+up3N1QnEj+H3JAqmU+g Sf9IRDyMNMf6hhzOQJxP8eNm3Cs0IojLFe/U59/cpUfSDeQyPuprnOteYLidbSBRRObx pQkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=aa6iExGIwghZDn1ZohvQXbuCEdzySOldf4bort+1eX4=; b=NkhEIbJCkjeasABT+oS5okLNQufuJoN07P5CO1Cs2zgV1+ssumYu3vl92himWLeQ2L 9KjRiS/3S1I+JHV9kx06RjoOIwYK4Kpw7ExIfojTGnLZvQQQ2W3g5sgUYdBWCbDa9T8r H2hT4CkzBks6/JyYI/oVPo0OzoD/N3indsbdKHb0b61QNvKUrCZio1f5fxE51g+g77YL yysHuR6Q5PryScpaqQUA9Nhy5DYqMu/glq2TCpUk5WKfH/hRwAMes6iGttBAV7ktXBDQ zaoO1Ge8+5RlRsFPpjU0JDQKKVurCM9hOeIJhakEF/J+8eq5HARcLZn7oXTHmpLDQCdD oyVA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=8XjWeMn3; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id i13si3522891qta.371.2019.04.02.21.24.15 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:24:15 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=8XjWeMn3; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id F3F2921BCD; Wed, 3 Apr 2019 00:24:14 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:24:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=aa6iExGIwghZDn1ZohvQXbuCEdzySOldf4bort+1eX4=; b=8XjWeMn3 stH473eDauTmxcBAVjsdZ6VLe07BfcTO+2jHtP6gWq0xcnjBPRi3hGnlBM9khyi4 x7rcfhtn6on7u25FsA2urq4JCHEBepYJkczrl/3vlZ48x4/DSRQKqoEWbBnBvVfJ MxfOOHWtw0PZu+4cT57honGKLamY1JbCpj+a27p+axoBMaxq5f4bSlAr2TeE93n/ U5cd+18ck6m9ASBOCRNlBOQYvXX19OfKV4J3f9gfs/p/Ykio4ZEep0Nd/8CLNKMu SmIyQa15UNhCGj7disyal3eeyG8Ig8iFK8sUvsUwB+8kAaMt75jfZBoxCvTOLV4w pZs8GJCUsqb2jg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeduud X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 22EAF100E5; Wed, 3 Apr 2019 00:24:07 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 12/14] slub: Enable balancing slabs across nodes Date: Wed, 3 Apr 2019 15:21:25 +1100 Message-Id: <20190403042127.18755-13-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We have just implemented Slab Movable Objects (SMO). On NUMA systems slabs can become unbalanced i.e. many slabs on one node while other nodes have few slabs. Using SMO we can balance the slabs across all the nodes. The algorithm used is as follows: 1. Move all objects to node 0 (this has the effect of defragmenting the cache). 2. Calculate the desired number of slabs for each node (this is done using the approximation nr_slabs / nr_nodes). 3. Loop over the nodes moving the desired number of slabs from node 0 to the node. Feature is conditionally built in with CONFIG_SMO_NODE, this is because we need the full list (we enable SLUB_DEBUG to get this). Future version may separate final list out of SLUB_DEBUG. Expose this functionality to userspace via a sysfs entry. Add sysfs entry: /sysfs/kernel/slab//balance Write of '1' to this file triggers balance, no other value accepted. This feature relies on SMO being enable for the cache, this is done with a call to, after the isolate/migrate functions have been defined. kmem_cache_setup_mobility(s, isolate, migrate) Signed-off-by: Tobin C. Harding --- mm/slub.c | 120 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index e4f3dde443f5..a5c48c41d72b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4583,6 +4583,109 @@ static unsigned long kmem_cache_move_to_node(struct kmem_cache *s, int node) return left; } + +/* + * kmem_cache_move_slabs() - Attempt to move @num slabs to target_node, + * @s: The cache we are working on. + * @node: The node to move objects from. + * @target_node: The node to move objects to. + * @num: The number of slabs to move. + * + * Attempts to move @num slabs from @node to @target_node. This is done + * by migrating objects from slabs on the full_list. + * + * Return: The number of slabs moved or error code. + */ +static long kmem_cache_move_slabs(struct kmem_cache *s, + int node, int target_node, long num) +{ + struct kmem_cache_node *n = get_node(s, node); + LIST_HEAD(move_list); + struct page *page, *page2; + unsigned long flags; + void **scratch; + long done = 0; + + if (node == target_node) + return -EINVAL; + + scratch = alloc_scratch(s); + if (!scratch) + return -ENOMEM; + + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &n->full, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + list_move(&page->lru, &move_list); + page->frozen = 1; + slab_unlock(page); + + if (++done >= num) + break; + } + spin_unlock_irqrestore(&n->list_lock, flags); + + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + /* + * This is best effort only, if slab still has + * objects just put it back on the partial list. + */ + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); + + return done; +} + +/* + * kmem_cache_balance_nodes() - Balance slabs across nodes. + * @s: The cache we are working on. + */ +static void kmem_cache_balance_nodes(struct kmem_cache *s) +{ + struct kmem_cache_node *n = get_node(s, 0); + unsigned long desired_nr_slabs_per_node; + unsigned long nr_slabs; + int nr_nodes = 0; + int nid; + + (void)kmem_cache_move_to_node(s, 0); + + for_each_node_state(nid, N_NORMAL_MEMORY) + nr_nodes++; + + nr_slabs = atomic_long_read(&n->nr_slabs); + desired_nr_slabs_per_node = nr_slabs / nr_nodes; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == 0) + continue; + + kmem_cache_move_slabs(s, 0, nid, desired_nr_slabs_per_node); + } +} #endif /** @@ -5847,6 +5950,22 @@ static ssize_t move_store(struct kmem_cache *s, const char *buf, size_t length) return length; } SLAB_ATTR(move); + +static ssize_t balance_show(struct kmem_cache *s, char *buf) +{ + return 0; +} + +static ssize_t balance_store(struct kmem_cache *s, + const char *buf, size_t length) +{ + if (buf[0] == '1') + kmem_cache_balance_nodes(s); + else + return -EINVAL; + return length; +} +SLAB_ATTR(balance); #endif /* CONFIG_SMO_NODE */ #ifdef CONFIG_NUMA @@ -5975,6 +6094,7 @@ static struct attribute *slab_attrs[] = { &shrink_attr.attr, #ifdef CONFIG_SMO_NODE &move_attr.attr, + &balance_attr.attr, #endif &slabs_cpu_partial_attr.attr, #ifdef CONFIG_SLUB_DEBUG From patchwork Wed Apr 3 04:21:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882765 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E46EC17E1 for ; Wed, 3 Apr 2019 04:24:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC22A287A6 for ; Wed, 3 Apr 2019 04:24:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C0253289BF; Wed, 3 Apr 2019 04:24:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 480DC287A6 for ; Wed, 3 Apr 2019 04:24:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3FB006B0275; Wed, 3 Apr 2019 00:24:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3AB5E6B027E; Wed, 3 Apr 2019 00:24:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2739B6B027F; Wed, 3 Apr 2019 00:24:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 094BE6B0275 for ; Wed, 3 Apr 2019 00:24:24 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id f196so13644635qke.4 for ; Tue, 02 Apr 2019 21:24:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=bbV+w3gF2LYD1FivN0kmZexmR5wyxe6wR2gkjGxFRYI=; b=iuhwCeLcFSW1+bHBvoIRPw2fy+ZKtJ7rEA2a7Ehg8fR44VtkyCVeIyNPWS9L5AYmVf xfApj8zrTNZABrug8v/zt3hdTH2LiGcsldy4Bf5KUvnB9QyniY7+t7N/SLcC42L5URUJ Be3Iwrt1BbjRCJyMeMDFTursTL23gc4v69OlI91jg5q850ieoUACezsqj2Nb+B4v/yM4 0VwO6cDe6hKpYSGjM1ttEMPIkeco6WNBeu42HYgVbJhoN1Tl930WEX6uqPuvckZcHML1 MyDQx4olbFq6x46903TQT6ZvQStmEGnu8gHVeOfdNbhqYvbYWoADj/KuCFdV+A8OtlBN I1hQ== X-Gm-Message-State: APjAAAX1p/jcaAi7rAFyIetJVvYsy0uBTeGiruj3XqVjqWy6z5qk7Ssc GdPfaQxwPR3BT+1JsG59cZaW+rZn21F0VoQoTko+X0GSVaqeIYMyvPSHPnC9av+axeir29+sZ/n YD5l4vZ93a9caUxlj1TPf4A+j3E5Y7vuzJpNDDsdTeo97j6R3paUW8Crg6h0Np7k= X-Received: by 2002:a0c:d25a:: with SMTP id o26mr62114176qvh.78.1554265463802; Tue, 02 Apr 2019 21:24:23 -0700 (PDT) X-Google-Smtp-Source: APXvYqxMx+FSty8XJ+f6wtel8o/wODmPu3LTtDn0fTyGBWceT7VIhixxXWiuN+BirQoX4WVCD77r X-Received: by 2002:a0c:d25a:: with SMTP id o26mr62114137qvh.78.1554265462736; Tue, 02 Apr 2019 21:24:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265462; cv=none; d=google.com; s=arc-20160816; b=A1YAZAEr3djhYO/s8CxwctM0AdWYI/rcFcg40kyAY0jXZVOhqlgkpIDyhc2CbQugtm 3OWOQFpJWW1NqjWozrJhFDgKVV5/pDwhxgPVDihojCm5JDp2e7q9xH8bX/LS8UHOVYAD bxG5TNEhBe1RcgLOw8ETyncuxkodkZd4fzrwsPjYDKuwVCfjzQiYp8UIWPggPNBZ+LM1 GjSzssVAsXA5NXSR+BN5K2FtpnwtW06LEpZcGvOXx1U5cQ4nLM/B9LMthCXTf+rO+t7k weqsK/N2Y8ixUArXH2dXGJVsP04TsRMSJjWompMBgDcP9gLPTI/RDyzmewsIfxbp+225 mZNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=bbV+w3gF2LYD1FivN0kmZexmR5wyxe6wR2gkjGxFRYI=; b=F1gKre+tZTexyUWS0SZ4kZ+VEMjtq2fdDlPd3q5cVfyd1TeocAJzWPqnuuKqIjdpCq +J6TCsYbnEjJ7Ck5JO85zGUrnsz4lodOF3qiR5hmJM34bQJB/26AW85HUA4clraoQYYf g2qsYEe5rvn5RK/7rQO9CDUco3ozH6I2scs0fSFwgxQs1LgVRgNvIiyv0+BwhHUDhjdu 6pTTfHQMdGWcnwi2oFM+3llkL5/t20pVc7AhinOPCs4FGf+ZOd33rVtqnMK9X8X8RuBG KwiIGmuDz4vxoAw9JO8fkGhvxRRz5TRSNMaOpyJQ6YxGVERILveVWu4VHmBxO+sEVwZx z+Nw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=A2qei93U; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id p9si2375945qkj.232.2019.04.02.21.24.22 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:24:22 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=A2qei93U; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 79D1021F93; Wed, 3 Apr 2019 00:24:22 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:24:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=bbV+w3gF2LYD1FivN0kmZexmR5wyxe6wR2gkjGxFRYI=; b=A2qei93U HU48t5f6nD+xKkxRcoxHrLa8FeTIc145hRYP6nIdf5yeW5Zh1r9KqwlDsYVUWOg0 wSndlC0Vr0TaWNSFGBsHApW+etgX/1SU/sgVCBjNp+JmkFvfLouRJ0hXN6DN1XXj cyiPzg1vPdxP6JhKQUnxEMMwJoJJCZa+YTEJQGaQaoB3J8eQ6Ghlx2PVy6Oi+lKI Jan3g6XBb0xIBNe1y9V6kSiajNCgmekjCpmw2E4+7S9L5AJQOXoI8xp9QQLrs1st bbj3mmvbN36dWYpHFUCPnY8100JoI/4MMxfQlz6CR2rqRfQHD7VbaDLR1uZASy2i oIB3q1xqQ7hy2w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeduvd X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 8369E1030F; Wed, 3 Apr 2019 00:24:15 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 13/14] dcache: Provide a dentry constructor Date: Wed, 3 Apr 2019 15:21:26 +1100 Message-Id: <20190403042127.18755-14-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In order to support object migration on the dentry cache we need to have a determined object state at all times. Without a constructor the object would have a random state after allocation. Provide a dentry constructor. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index aac41adf4743..606844ad5171 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -1603,6 +1603,22 @@ void d_invalidate(struct dentry *dentry) } EXPORT_SYMBOL(d_invalidate); +static void dcache_ctor(void *p) +{ + struct dentry *dentry = p; + + dentry->d_lockref.count = 0; + dentry->d_inode = NULL; + + spin_lock_init(&dentry->d_lock); + + INIT_HLIST_BL_NODE(&dentry->d_hash); + INIT_LIST_HEAD(&dentry->d_lru); + INIT_LIST_HEAD(&dentry->d_subdirs); + INIT_HLIST_NODE(&dentry->d_u.d_alias); + INIT_LIST_HEAD(&dentry->d_child); +} + /** * __d_alloc - allocate a dcache entry * @sb: filesystem it will belong to @@ -1658,7 +1674,7 @@ struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) dentry->d_lockref.count = 1; dentry->d_flags = 0; - spin_lock_init(&dentry->d_lock); + seqcount_init(&dentry->d_seq); dentry->d_inode = NULL; dentry->d_parent = dentry; @@ -3091,14 +3107,17 @@ static void __init dcache_init_early(void) static void __init dcache_init(void) { - /* - * A constructor could be added for stable state like the lists, - * but it is probably not worth it because of the cache nature - * of the dcache. - */ - dentry_cache = KMEM_CACHE_USERCOPY(dentry, - SLAB_RECLAIM_ACCOUNT|SLAB_PANIC|SLAB_MEM_SPREAD|SLAB_ACCOUNT, - d_iname); + slab_flags_t flags = + SLAB_RECLAIM_ACCOUNT | SLAB_PANIC | SLAB_MEM_SPREAD | SLAB_ACCOUNT; + + dentry_cache = + kmem_cache_create_usercopy("dentry", + sizeof(struct dentry), + __alignof__(struct dentry), + flags, + offsetof(struct dentry, d_iname), + sizeof_field(struct dentry, d_iname), + dcache_ctor); /* Hash may have been set up in dcache_init_early */ if (!hashdist) From patchwork Wed Apr 3 04:21:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882767 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BEAAD1575 for ; Wed, 3 Apr 2019 04:24:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A2E5227F93 for ; Wed, 3 Apr 2019 04:24:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8F09D284C7; Wed, 3 Apr 2019 04:24:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE6D3287A6 for ; Wed, 3 Apr 2019 04:24:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DBF0E6B027F; Wed, 3 Apr 2019 00:24:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D6F586B0280; Wed, 3 Apr 2019 00:24:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C37CE6B0281; Wed, 3 Apr 2019 00:24:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id A40DD6B027F for ; Wed, 3 Apr 2019 00:24:31 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id z34so15667806qtz.14 for ; Tue, 02 Apr 2019 21:24:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=3CgZchYLd16ww3/LiD/mAfG7GXA+RJSG10+pHsLIDks=; b=nQlYBqot0kpcuzlZ+bjFOkqxaZvoxAJ9ox2kDDoAnxiG6Q9t0rm6r+MUWH7Q9wIwXn 1MkaRrEmGgWhD/AMWjGFqKWlhtpAPSkrTZMARMfRfYFDR7EN3w2WZzBm4s064SMtCiY8 eiykS6gE8nXFZwapk71vkqBhJupxt7Gn3Gp8z2hATDduevGCExqdo9xGGFr9NbaAkmzc Mnot/9wY3vYlV76uvsfPrni7bMgWteBv4eWIYiHcNX6qwn6sKJdUahgBtSfpj0RxWufG LZIRqdR1fyH4T1NFydIQCrO8Ek5l6TBcTXfTxx5mnwL8ot+AqBizMEhjbTZJBQVG83xs DO6A== X-Gm-Message-State: APjAAAXAPq+WBji3MnPM8xIhNhjnbDZzSTj/30gudzuSnbljyI6xbUF4 od2op63+19fUbPqBM+ODezC4aYK1DepDJZQRUaJb2vNUeHxKYCkJZnsqyI0GNPE8UuLAb4OYwzy +6+AJLbyyUlVjAGofqlfr0rWsvbo7nn0QmX9T1ADGkSEZLNGeHMyx+HV3goiKTfs= X-Received: by 2002:a0c:81e2:: with SMTP id 31mr25271444qve.179.1554265471416; Tue, 02 Apr 2019 21:24:31 -0700 (PDT) X-Google-Smtp-Source: APXvYqzykPWLxF7/qOPnYO5FBUcMWiT5GC2F7KFuQj+bc6b4N8oW8GZ4o1ny0QC8Ld/iuJYCD0sJ X-Received: by 2002:a0c:81e2:: with SMTP id 31mr25271396qve.179.1554265470185; Tue, 02 Apr 2019 21:24:30 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265470; cv=none; d=google.com; s=arc-20160816; b=OxPgaQoCvijFjP+XskRWfcQRqaxbWCUCywlMJOezJLvVKizxuDWnzh17bySO0hqk+J S7cLk+8PCYqLLmfLuuL/HVjoDfsLCSirwgOJEWNNSMP8PjfXrfqJDDQRrAAU2d7bPYoW aqM7q9NiX494alKUlpPU7xniuGs862lenljHOUanhV99yNNeLc9v3hlHD8k41QjOM41/ g+UEvX5vNGtlY5wOPs/oWPjhhPIeGRp6Nb4wd1OiT3kWW2/uB1Vx8XHj88J6M3r2P8JR JFQYK1lqbeSLTJZ9TOsXnvL74cGdK4/O20o6junIhxpFQgQAewnXD8HZDfEzQg42Ejum 0/cg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=3CgZchYLd16ww3/LiD/mAfG7GXA+RJSG10+pHsLIDks=; b=dCiJAFjrzm7ii1fvcBX0mZuyB6jyXqDy+ee8xgnsh0Q/3M2AUqr29+RGbVjiIMKcJz Aw/85Y+DAxecHOMKtLGBvdJcUuV1iGJZ9I6+jMrRQuTZLJ/+Dpc0JsK08I/DoJDjbdbC 4Oh4QlNY9EMcONJ8VWktqAz4cgYI/Op5zI6Iz9eN03y2+BJkwqpcefeworH8I2bd6zbP soPVppYNBhamyTjU7MXbK06vk7xj8Ja7K0C0XRRQ/y0uteXXh0lRKnY433ZJrZcznKrE uFdhHcGMVDMWtfpQRbn5ZVwrU+kIFCm/vzxVD8zRb/Tb3Fd8s+ejQgsRZ43apS1FJ6Dt tAOg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=wY4kIaUF; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id n24si1690641qkk.23.2019.04.02.21.24.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:24:30 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=wY4kIaUF; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id E23B621FAE; Wed, 3 Apr 2019 00:24:29 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:24:29 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=3CgZchYLd16ww3/LiD/mAfG7GXA+RJSG10+pHsLIDks=; b=wY4kIaUF bKeyoZ1pX1nCWrBGHGHId2HQJ7fpODFRyw9W9hbVgEBM2F+znekZY9/Ya6bpkQLx g4bxXkC0jILJxfUFw+Xp/nP9yTQgozkM4b4uJR7dSeJn7Du55PXTCh12806jtLbw ZZtzaSMa5mrjyPZji51A8vd+/ku4ZVL5Qd4QTYWu6LVgY3XDelhAATeQIwGLJEeb QLOnXlctXLmYy9UVB2VQXXHy3Vce14hsYi8KLRqRXNOdpfUzNHOb/opiGiwyMH4x NkPSYl8X0PHNA6n6iOzmg96i85YHVOD3FW9jf9WsUSTJx9TKvaAriKhZyw0pb5db fRyyTYkfF5tEDg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpedufe X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 0A2BB10319; Wed, 3 Apr 2019 00:24:22 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 14/14] dcache: Implement object migration Date: Wed, 3 Apr 2019 15:21:27 +1100 Message-Id: <20190403042127.18755-15-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The dentry slab cache is susceptible to internal fragmentation. Now that we have Slab Movable Objects we can defragment the dcache. Object migration is only possible for dentry objects that are not currently referenced by anyone, i.e. we are using the object migration infrastructure to free unused dentries. Implement isolate and migrate functions for the dentry slab cache. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+) diff --git a/fs/dcache.c b/fs/dcache.c index 606844ad5171..4387715b7ebb 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -30,6 +30,7 @@ #include #include #include +#include #include "internal.h" #include "mount.h" @@ -3074,6 +3075,90 @@ void d_tmpfile(struct dentry *dentry, struct inode *inode) } EXPORT_SYMBOL(d_tmpfile); +/* + * d_isolate() - Dentry isolation callback function. + * @s: The dentry cache. + * @v: Vector of pointers to the objects to migrate. + * @nr: Number of objects in @v. + * + * The slab allocator is holding off frees. We can safely examine + * the object without the danger of it vanishing from under us. + */ +static void *d_isolate(struct kmem_cache *s, void **v, int nr) +{ + struct dentry *dentry; + int i; + + for (i = 0; i < nr; i++) { + dentry = v[i]; + spin_lock(&dentry->d_lock); + /* + * Three sorts of dentries cannot be reclaimed: + * + * 1. dentries that are in the process of being allocated + * or being freed. In that case the dentry is neither + * on the LRU nor hashed. + * + * 2. Fake hashed entries as used for anonymous dentries + * and pipe I/O. The fake hashed entries have d_flags + * set to indicate a hashed entry. However, the + * d_hash field indicates that the entry is not hashed. + * + * 3. dentries that have a backing store that is not + * writable. This is true for tmpsfs and other in + * memory filesystems. Removing dentries from them + * would loose dentries for good. + */ + if ((d_unhashed(dentry) && list_empty(&dentry->d_lru)) || + (!d_unhashed(dentry) && hlist_bl_unhashed(&dentry->d_hash)) || + (dentry->d_inode && + !mapping_cap_writeback_dirty(dentry->d_inode->i_mapping))) { + /* Ignore this dentry */ + v[i] = NULL; + } else { + __dget_dlock(dentry); + } + spin_unlock(&dentry->d_lock); + } + return NULL; /* No need for private data */ +} + +/* + * d_migrate() - Dentry migration callback function. + * @s: The dentry cache. + * @v: Vector of pointers to the objects to migrate. + * @nr: Number of objects in @v. + * @node: The NUMA node where new object should be allocated. + * @private: Returned by d_isolate() (currently %NULL). + * + * Slab has dropped all the locks. Get rid of the refcount obtained + * earlier and also free the object. + */ +static void d_migrate(struct kmem_cache *s, void **v, int nr, + int node, void *_unused) +{ + struct dentry *dentry; + int i; + + for (i = 0; i < nr; i++) { + dentry = v[i]; + if (dentry) + d_invalidate(dentry); + } + + for (i = 0; i < nr; i++) { + dentry = v[i]; + if (dentry) + dput(dentry); + } + + /* + * dentries are freed using RCU so we need to wait until RCU + * operations are complete. + */ + synchronize_rcu(); +} + static __initdata unsigned long dhash_entries; static int __init set_dhash_entries(char *str) { @@ -3119,6 +3204,8 @@ static void __init dcache_init(void) sizeof_field(struct dentry, d_iname), dcache_ctor); + kmem_cache_setup_mobility(dentry_cache, d_isolate, d_migrate); + /* Hash may have been set up in dcache_init_early */ if (!hashdist) return;