From patchwork Tue Apr 30 03:07:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922729 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 402411390 for ; Tue, 30 Apr 2019 03:08:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2BFCA28994 for ; Tue, 30 Apr 2019 03:08:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1B54A289C9; Tue, 30 Apr 2019 03:08:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4453128994 for ; Tue, 30 Apr 2019 03:08:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 559F26B000C; Mon, 29 Apr 2019 23:08:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 50B5F6B000D; Mon, 29 Apr 2019 23:08:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D4206B000E; Mon, 29 Apr 2019 23:08:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id 19E7B6B000C for ; Mon, 29 Apr 2019 23:08:49 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id t63so10744220qkh.0 for ; Mon, 29 Apr 2019 20:08:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ToWY0R45y8BHjmAdWxteY+oDuIkdik3lI8PSNGM4h2M=; b=jwLvbDBaFQvvUndilbodu3iFwlV8jE6/Yu8jikUktOSAmMOpTG274ihzubwceppTPf zrUcu+69idXmaZin0uDIOOLEHx6dVLoY9N+xyDleljuPxYtT/lMkU1wL8jzfcmJC0mD7 hvEdxaxvkLmoGRdwW+j1irKKghPC6LHLZpp4srkXXu1pQsDt99rbQa07+eCyYpjDHG3i Np43xsWVrUy3c0sPiSRDkF3wZ9bvWUwEsHBKmfmbcYzeYlG9b4BfXtC+9ao6BRnBwET4 2CCSHmTfPHEbGDyoMKE+KHDSTlej6CX0/Um8XmPnLhv4Trn8bTSRqh8+hSGTSugeWUaQ 6RrQ== X-Gm-Message-State: APjAAAVP+ojZGpGmKTHu1KIBbsbQvccOt4iuSNUkkeBdAWw2VTUw4YGm riNDfrSMSMdGBR+y3UW8ahHgJgFJoqhr75tHPnxP8FLLxh8t1qnd/EB4nDT8jUfqkc9CroAXcSD aYmBpuElSg4eG4tb3ow4VfsoCFHKj2YkESNmGZhLohvSkOXfKaaAbvQXxVxMuZ2c= X-Received: by 2002:ac8:2924:: with SMTP id y33mr51052141qty.212.1556593728804; Mon, 29 Apr 2019 20:08:48 -0700 (PDT) X-Google-Smtp-Source: APXvYqzKf0q4yVw7e0QlLna2vAcNfioZ78G5QFE000vvhHdjLtSM5TYzbCQasLPoYVVyQ1dvY70A X-Received: by 2002:ac8:2924:: with SMTP id y33mr51052076qty.212.1556593727395; Mon, 29 Apr 2019 20:08:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593727; cv=none; d=google.com; s=arc-20160816; b=vCoRfUMPTAHSjhCMV78O/eB+7BUQSEuuv9MKLP0b258WaPNuaiauBr7bOHM2J2xGAO wdkJBqqodiId+Q6Xj/UzMcbhz1jSV4fDzu1AJMH8gc1kpqFnlYDhYwM2stLnS/CO64eS N67h5tbzCJN3Sq1M905flsku9OzSLtpUyB+97ZHQZjIbQNlJWL39m1cxMFkuhaFjgCLH jT9MS1IbtcvS9qjcft1gPICRjYcAKb0m0zAK+8B4BgXXfkO2mRhqidtz2xiVPoxf9ZIq O4oGBgZkyFYNl3iRZuVmP96YDqiSApEo8aFbNZtrtopTIosRJcc7XXQEGrv7CXnQ63VR NxUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=ToWY0R45y8BHjmAdWxteY+oDuIkdik3lI8PSNGM4h2M=; b=bkzV8ptStVE1cyt9dLd9UgBFNVKhMC+wZBXrI2Pb+om6SA1G33B+9/q4fERXm3K1TA UGJUv74ssuWtggKkoHS8dQTHb3oWRlMyvc+Mye+AH+PV8MX0sU2pxveQQPt/ntLO1uZS 5TO9Ndw/3zHGdfXn7xsMccPRgWIiqR3dy8HFhZ3426sL5eIkx+NrlYf21qMF383r9isw Xl95PccSQih5/j1ZoiwHuN3cqYX4oFcvmEF2YuVYkQRh6jL/0w3angTqLF+Rcq1NtGoE pUg1EH0sHmXj75B5kLYLyOwC7/gSs9incK3pUNpr9TFFcHmhbCx4VZoHnpd/Jgl75nbT XbOw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=XXcBFbLy; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id t5si6308916qvm.17.2019.04.29.20.08.47 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:08:47 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=XXcBFbLy; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 22555DA1D; Mon, 29 Apr 2019 23:08:47 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:08:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=ToWY0R45y8BHjmAdWxteY+oDuIkdik3lI8PSNGM4h2M=; b=XXcBFbLy 7CsIB54OmO3t9u36Gqt/SFAsTZ2sSbQXaThBA3qa/du9X8VBd4G2r0OkL+C8Yjqg pXOwH9IBjsUEOkTfm+GC+7hImvu/LeaTwLudmHYjWY3KmLukO0hjusWqEsjcgoY2 7Lq95c8JGCepwOTAIp36JhIFfI7qLU2PVDwR1B0Yh/2zZmjgCDlqIgcYs11R9nwn IpZ/X9l/DQysA1F926YMUIbIlgP6fFgSvStvNdAiRXvyNF9w2gxPwXgaI45b/Qn+ Brg7mr58C6/CUsEXAFasj7ezCLUqIwh+1dIUGjh3Cii8e2LwfbNGOR+GP2spdjh7 VR3DHONTodgAlw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 58225103C9; Mon, 29 Apr 2019 23:08:39 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 01/15] slub: Add isolate() and migrate() methods Date: Tue, 30 Apr 2019 13:07:32 +1000 Message-Id: <20190430030746.26102-2-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the two methods needed for moving objects and enable the display of the callbacks via the /sys/kernel/slab interface. Add documentation explaining the use of these methods and the prototypes for slab.h. Add functions to setup the callbacks method for a slab cache. Add empty functions for SLAB/SLOB. The API is generic so it could be theoretically implemented for these allocators as well. Change sysfs 'ctor' field to be 'ops' to contain all the callback operations defined for a slab cache. Display the existing 'ctor' callback in the ops fields contents along with 'isolate' and 'migrate' callbacks. Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- include/linux/slab.h | 70 ++++++++++++++++++++++++++++++++++++++++ include/linux/slub_def.h | 3 ++ mm/slub.c | 59 +++++++++++++++++++++++++++++---- 3 files changed, 126 insertions(+), 6 deletions(-) diff --git a/include/linux/slab.h b/include/linux/slab.h index 9449b19c5f10..886fc130334d 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -154,6 +154,76 @@ void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); void memcg_destroy_kmem_caches(struct mem_cgroup *); +/* + * Function prototypes passed to kmem_cache_setup_mobility() to enable + * mobile objects and targeted reclaim in slab caches. + */ + +/** + * typedef kmem_cache_isolate_func - Object migration callback function. + * @s: The cache we are working on. + * @ptr: Pointer to an array of pointers to the objects to isolate. + * @nr: Number of objects in @ptr array. + * + * The purpose of kmem_cache_isolate_func() is to pin each object so that + * they cannot be freed until kmem_cache_migrate_func() has processed + * them. This may be accomplished by increasing the refcount or setting + * a flag. + * + * The object pointer array passed is also passed to + * kmem_cache_migrate_func(). The function may remove objects from the + * array by setting pointers to %NULL. This is useful if we can + * determine that an object is being freed because + * kmem_cache_isolate_func() was called when the subsystem was calling + * kmem_cache_free(). In that case it is not necessary to increase the + * refcount or specially mark the object because the release of the slab + * lock will lead to the immediate freeing of the object. + * + * Context: Called with locks held so that the slab objects cannot be + * freed. We are in an atomic context and no slab operations + * may be performed. + * Return: A pointer that is passed to the migrate function. If any + * objects cannot be touched at this point then the pointer may + * indicate a failure and then the migration function can simply + * remove the references that were already obtained. The private + * data could be used to track the objects that were already pinned. + */ +typedef void *kmem_cache_isolate_func(struct kmem_cache *s, void **ptr, int nr); + +/** + * typedef kmem_cache_migrate_func - Object migration callback function. + * @s: The cache we are working on. + * @ptr: Pointer to an array of pointers to the objects to migrate. + * @nr: Number of objects in @ptr array. + * @node: The NUMA node where the object should be allocated. + * @private: The pointer returned by kmem_cache_isolate_func(). + * + * This function is responsible for migrating objects. Typically, for + * each object in the input array you will want to allocate an new + * object, copy the original object, update any pointers, and free the + * old object. + * + * After this function returns all pointers to the old object should now + * point to the new object. + * + * Context: Called with no locks held and interrupts enabled. Sleeping + * is possible. Any operation may be performed. + */ +typedef void kmem_cache_migrate_func(struct kmem_cache *s, void **ptr, + int nr, int node, void *private); + +/* + * kmem_cache_setup_mobility() is used to setup callbacks for a slab cache. + */ +#ifdef CONFIG_SLUB +void kmem_cache_setup_mobility(struct kmem_cache *, kmem_cache_isolate_func, + kmem_cache_migrate_func); +#else +static inline void +kmem_cache_setup_mobility(struct kmem_cache *s, kmem_cache_isolate_func isolate, + kmem_cache_migrate_func migrate) {} +#endif + /* * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above. diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index d2153789bd9f..2879a2f5f8eb 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -99,6 +99,9 @@ struct kmem_cache { gfp_t allocflags; /* gfp flags to use on each alloc */ int refcount; /* Refcount for slab cache destroy */ void (*ctor)(void *); + kmem_cache_isolate_func *isolate; + kmem_cache_migrate_func *migrate; + unsigned int inuse; /* Offset to metadata */ unsigned int align; /* Alignment */ unsigned int red_left_pad; /* Left redzone padding size */ diff --git a/mm/slub.c b/mm/slub.c index d30ede89f4a6..ae44d640b8c1 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4326,6 +4326,33 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return err; } +void kmem_cache_setup_mobility(struct kmem_cache *s, + kmem_cache_isolate_func isolate, + kmem_cache_migrate_func migrate) +{ + /* + * Mobile objects must have a ctor otherwise the object may be + * in an undefined state on allocation. Since the object may + * need to be inspected by the migration function at any time + * after allocation we must ensure that the object always has a + * defined state. + */ + if (!s->ctor) { + pr_err("%s: require constructor to setup mobility\n", s->name); + return; + } + + s->isolate = isolate; + s->migrate = migrate; + + /* + * Sadly serialization requirements currently mean that we have + * to disable fast cmpxchg based processing. + */ + s->flags &= ~__CMPXCHG_DOUBLE; +} +EXPORT_SYMBOL(kmem_cache_setup_mobility); + void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller) { struct kmem_cache *s; @@ -5010,13 +5037,33 @@ static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf, } SLAB_ATTR(cpu_partial); -static ssize_t ctor_show(struct kmem_cache *s, char *buf) +static int op_show(char *buf, const char *txt, unsigned long addr) { - if (!s->ctor) - return 0; - return sprintf(buf, "%pS\n", s->ctor); + int x = 0; + + x += sprintf(buf, "%s : ", txt); + x += sprint_symbol(buf + x, addr); + x += sprintf(buf + x, "\n"); + + return x; +} + +static ssize_t ops_show(struct kmem_cache *s, char *buf) +{ + int x = 0; + + if (s->ctor) + x += op_show(buf + x, "ctor", (unsigned long)s->ctor); + + if (s->isolate) + x += op_show(buf + x, "isolate", (unsigned long)s->isolate); + + if (s->migrate) + x += op_show(buf + x, "migrate", (unsigned long)s->migrate); + + return x; } -SLAB_ATTR_RO(ctor); +SLAB_ATTR_RO(ops); static ssize_t aliases_show(struct kmem_cache *s, char *buf) { @@ -5429,7 +5476,7 @@ static struct attribute *slab_attrs[] = { &objects_partial_attr.attr, &partial_attr.attr, &cpu_slabs_attr.attr, - &ctor_attr.attr, + &ops_attr.attr, &aliases_attr.attr, &align_attr.attr, &hwcache_align_attr.attr, From patchwork Tue Apr 30 03:07:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922735 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D0A991390 for ; Tue, 30 Apr 2019 03:08:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C11C528994 for ; Tue, 30 Apr 2019 03:08:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B49F428A4E; Tue, 30 Apr 2019 03:08:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2172128994 for ; Tue, 30 Apr 2019 03:08:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0030A6B0010; Mon, 29 Apr 2019 23:08:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EF4FF6B0266; Mon, 29 Apr 2019 23:08:56 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBF0B6B0269; Mon, 29 Apr 2019 23:08:56 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id BC28B6B0010 for ; Mon, 29 Apr 2019 23:08:56 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id s70so10745304qka.1 for ; Mon, 29 Apr 2019 20:08:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=bxV5jBaXvA/LKL0qOR0ZLSK5IIfdrfWAd0nyjsopfiM=; b=l4H1Vio9dCuLy+Akq/ieMy4MKL2rJ+5UoWzC5WfdsEJvSDlnXbUYgI++e9rR0/Ufqw SzZ/8WNy0/CVfpaHCGkxh/biyTczlN3+G1Jk/U+aFZ/L7jCRBWsL0cqBXTgQ98dUBW1j iy584GAKreOFjSyDXYWilA0T/HnB9uuDYBG/4ueVnCMjbLc82P6qLlmb69BxGxdgXjza fzj5sHaLO+Z290BWHdyXIjcQYgtUqWOVIEktb89cv0Qx+moyjc1YpK9Mi/BFsmySqNPk AoMO2XO4d+d8w3+K+jEkkm1Iu0GmRq/keBsfyMlwUPNtKgp0wNX+oWnjIbWMOOS0fMDI 5lUg== X-Gm-Message-State: APjAAAWHzcE5g3nAjrppMJlXioYQ7rpt37pBGO78xflM9XP/lUIcig+8 OvkEbf4usFlHaXL/FrGH+SjUVlNL0TEHQ5tQKjmjb3FzoytJJBK1H2lgBm/dxKpNISRwQlc2Wax Y/iD4TNev6Vq+Bs7299nq32yNN/U/r+TkPwgWY+F3ow6XAZRIW/tpxcwHjTg1XTE= X-Received: by 2002:ac8:3868:: with SMTP id r37mr37307332qtb.199.1556593736482; Mon, 29 Apr 2019 20:08:56 -0700 (PDT) X-Google-Smtp-Source: APXvYqx4GLr9KDwaDihb781/M03Glr/pEg/NC+Mrj6OzmwBP7ur9qGjmTe26+PZdsiCpxq6jfEe8 X-Received: by 2002:ac8:3868:: with SMTP id r37mr37307289qtb.199.1556593735216; Mon, 29 Apr 2019 20:08:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593735; cv=none; d=google.com; s=arc-20160816; b=Ar0kEACWq4Ln5IRt+4B9cy5exCbh5CXRR0jDWSzFIzyVbuA2+alMmk4WwW0NNnTNvP 1AMthFnDCoicRm0lf8RCXGVSd1TMeUpUxMpm5BiybnHNJSG/zDq5KFglZgfZA1IpLhO5 mThUx2om3vP9i/G8UQcGq5/rB3dKSZTaD00OppvlExyp5bLyCnghUaxwyfEBF7spBdyo Foa1NcBrHCZ68ecCLMERahvNI17xjvyY7LrLO7lUBxlJ6Sgohh38nvVMI5sB1QHZmxxD GHY25nd4gppYeqGTLkEMLs9umlbH6tRhH7UgcuXXHafYHLVbrrPRAsjuaFlQXgKjvhfb 0tmA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=bxV5jBaXvA/LKL0qOR0ZLSK5IIfdrfWAd0nyjsopfiM=; b=EQpZXh3N2/aMYwa+lEPqNBhKvmjTiix9nXayxK6E1s1GXsQ/9aGSA1XwRMUt+9uyKr 71zkWwzRgJXyocH62BkBS2k0W73k5wWPm2LnYZ+1puYP1/cpJQptIJ2IZuJSkuSKQpqf KWM6Cs7XLvbUzEr1TYyIZ7i4A+5EcRQS0ErIm1f46egV2Ea5iTjssFz5+EVoYzo1VuZ1 y9jJOwBrA6VdCwj89XsIv1KRuXhzwTMefocthIvqlCYE8BnKINLLM6MhpdcEL0KHW5ee yShKyV7oIOX71L6t5XOgH2qacMzsbyrWukJ66IOSJ2VUvyHqhUEI6xX91c5fIYeA38sB aTHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=j6+4FHAE; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id p27si8572266qvc.166.2019.04.29.20.08.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:08:55 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=j6+4FHAE; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id EB66B5286; Mon, 29 Apr 2019 23:08:54 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:08:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=bxV5jBaXvA/LKL0qOR0ZLSK5IIfdrfWAd0nyjsopfiM=; b=j6+4FHAE FVM8Li58NZcWW6HLlE9bl9fVj9deVbeq+Jxjc8W0HH4/aQ6y4q8QepnkNhVe78pi /biDsaZbgcybTFU/OokAdYlEJiQ3BjySDRQjwsyDtEHzv3wZAFeqVoVqqFxiyrZw Fh0VSQXbrxiHBNEP02LAzOWkJWkulDgZ/Xg6sKV9S5ZLrf5h9ZfZcrZzLKJfcfFk guHwnmPVbrN3d96RTt3m6egGtUUq06Fmb9ypLapjBnRxm2tN9U6kaFAtNqf6qnPp ND1JWNDcqiGim5drGe0p40+254w9D8NgPXf7K+jIjDYecXZskVwftSTcrVoy1Xeb Vcl8ZCFCMSEmjw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedu X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 2CB2D103C8; Mon, 29 Apr 2019 23:08:46 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 02/15] tools/vm/slabinfo: Add support for -C and -M options Date: Tue, 30 Apr 2019 13:07:33 +1000 Message-Id: <20190430030746.26102-3-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP -C lists caches that use a ctor. -M lists caches that support object migration. Add command line options to show caches with a constructor and caches that are movable (i.e. have migrate function). Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 40 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 4 deletions(-) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index 73818f1b2ef8..cbfc56c44c2f 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -33,6 +33,7 @@ struct slabinfo { unsigned int hwcache_align, object_size, objs_per_slab; unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; + int movable, ctor; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; unsigned long free_fastpath, free_slowpath; @@ -67,6 +68,8 @@ int show_report; int show_alias; int show_slab; int skip_zero = 1; +int show_movable; +int show_ctor; int show_numa; int show_track; int show_first_alias; @@ -109,11 +112,13 @@ static void fatal(const char *x, ...) static void usage(void) { - printf("slabinfo 4/15/2011. (c) 2007 sgi/(c) 2011 Linux Foundation.\n\n" - "slabinfo [-aADefhilnosrStTvz1LXBU] [N=K] [-dafzput] [slab-regexp]\n" + printf("slabinfo 4/15/2017. (c) 2007 sgi/(c) 2011 Linux Foundation/(c) 2017 Jump Trading LLC.\n\n" + "slabinfo [-aACDefhilMnosrStTvz1LXBU] [N=K] [-dafzput] [slab-regexp]\n" + "-a|--aliases Show aliases\n" "-A|--activity Most active slabs first\n" "-B|--Bytes Show size in bytes\n" + "-C|--ctor Show slabs with ctors\n" "-D|--display-active Switch line format to activity\n" "-e|--empty Show empty slabs\n" "-f|--first-alias Show first alias\n" @@ -121,6 +126,7 @@ static void usage(void) "-i|--inverted Inverted list\n" "-l|--slabs Show slabs\n" "-L|--Loss Sort by loss\n" + "-M|--movable Show caches that support movable objects\n" "-n|--numa Show NUMA information\n" "-N|--lines=K Show the first K slabs\n" "-o|--ops Show kmem_cache_ops\n" @@ -588,6 +594,12 @@ static void slabcache(struct slabinfo *s) if (show_empty && s->slabs) return; + if (show_ctor && !s->ctor) + return; + + if (show_movable && !s->movable) + return; + if (sort_loss == 0) store_size(size_str, slab_size(s)); else @@ -602,6 +614,10 @@ static void slabcache(struct slabinfo *s) *p++ = '*'; if (s->cache_dma) *p++ = 'd'; + if (s->ctor) + *p++ = 'C'; + if (s->movable) + *p++ = 'M'; if (s->hwcache_align) *p++ = 'A'; if (s->poison) @@ -636,7 +652,8 @@ static void slabcache(struct slabinfo *s) printf("%-21s %8ld %7d %15s %14s %4d %1d %3ld %3ld %s\n", s->name, s->objects, s->object_size, size_str, dist_str, s->objs_per_slab, s->order, - s->slabs ? (s->partial * 100) / s->slabs : 100, + s->slabs ? (s->partial * 100) / + (s->slabs * s->objs_per_slab) : 100, s->slabs ? (s->objects * s->object_size * 100) / (s->slabs * (page_size << s->order)) : 100, flags); @@ -1256,6 +1273,13 @@ static void read_slab_dir(void) slab->alloc_node_mismatch = get_obj("alloc_node_mismatch"); slab->deactivate_bypass = get_obj("deactivate_bypass"); chdir(".."); + if (read_slab_obj(slab, "ops")) { + if (strstr(buffer, "ctor :")) + slab->ctor = 1; + if (strstr(buffer, "migrate :")) + slab->movable = 1; + } + if (slab->name[0] == ':') alias_targets++; slab++; @@ -1332,6 +1356,8 @@ static void xtotals(void) } struct option opts[] = { + { "ctor", no_argument, NULL, 'C' }, + { "movable", no_argument, NULL, 'M' }, { "aliases", no_argument, NULL, 'a' }, { "activity", no_argument, NULL, 'A' }, { "debug", optional_argument, NULL, 'd' }, @@ -1367,7 +1393,7 @@ int main(int argc, char *argv[]) page_size = getpagesize(); - while ((c = getopt_long(argc, argv, "aAd::Defhil1noprstvzTSN:LXBU", + while ((c = getopt_long(argc, argv, "aACd::Defhil1MnoprstvzTSN:LXBU", opts, NULL)) != -1) switch (c) { case '1': @@ -1376,6 +1402,9 @@ int main(int argc, char *argv[]) case 'a': show_alias = 1; break; + case 'C': + show_ctor = 1; + break; case 'A': sort_active = 1; break; @@ -1399,6 +1428,9 @@ int main(int argc, char *argv[]) case 'i': show_inverted = 1; break; + case 'M': + show_movable = 1; + break; case 'n': show_numa = 1; break; From patchwork Tue Apr 30 03:07:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922737 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 261541390 for ; Tue, 30 Apr 2019 03:09:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 14FE328994 for ; Tue, 30 Apr 2019 03:09:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 073B1289C9; Tue, 30 Apr 2019 03:09:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8A23528994 for ; Tue, 30 Apr 2019 03:09:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A89656B026A; Mon, 29 Apr 2019 23:09:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A44C36B026B; Mon, 29 Apr 2019 23:09:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 902E56B026C; Mon, 29 Apr 2019 23:09:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 708BF6B026A for ; Mon, 29 Apr 2019 23:09:04 -0400 (EDT) Received: by mail-qt1-f200.google.com with SMTP id c22so12160099qtk.10 for ; Mon, 29 Apr 2019 20:09:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=oAX6zCohmPiQ74kYACc2QgeImmemzleYDH4h+URdt/s=; b=oIV6bglmBOyBqcHCw089Q41pw/ClxRckPBtJcpOgAHO/+DZK78Em4YM8W05tqyL9ha OqyGFKbGmuoLYmPWagfyCI2DIi8s4Kcj/ODSWGBG0Sk3MswTdGc91v5xYEAGO+iHjRnW +FnbnBqAMp8EW1K96ERB1XvPbFdLdok00x+30qeQIlgpmiE1+Q0AqXKZVGzn+n+lcvvi 7PJz32hPdq01xtC/m4YAymHh2tlUyeYJTc5S1OgB7XeVoANY37crG5WJQ0TH85nz7zB7 MrOXqPjJfRR2A4BqTTAQQmiO7SewUKQGm3NN+34a6O3COTZQ/UTag96xmXqrokJ8dC/0 OQ5g== X-Gm-Message-State: APjAAAVriBqnRZV6BztpkaKj7EkxpaWCGa6GwRBI7QQiNz5yYhOBZfYk q6O4eN5b7uOpYolbh75d6GMaErJy+p1aNXY+ay84JV0hzUPXVuz6IlZadxlyBtM5oY44vpm0EfU HZ260BhVrECtgdB+iR36bGUBQCcaV/s2kzNGCjySj5j8eaX6/j498yReqvDqcQgA= X-Received: by 2002:aed:30c1:: with SMTP id 59mr50879357qtf.277.1556593744240; Mon, 29 Apr 2019 20:09:04 -0700 (PDT) X-Google-Smtp-Source: APXvYqwnxyaV3x3JVRK1NQqfxdSGgt8WwUbqxiVs1LYXRvCv9/1lx1j3XkZ7MFbP3eBLkt9JxRyT X-Received: by 2002:aed:30c1:: with SMTP id 59mr50879318qtf.277.1556593743326; Mon, 29 Apr 2019 20:09:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593743; cv=none; d=google.com; s=arc-20160816; b=hyf27IZuKIJ2dy45VpNWHeokFOLmDV1dm6QZpqO5wkt2gjrlNAtc+J9rALeDUfYB/q Orm3vDRfmuDjGTfCLle1QQ+GHa5nPqtXkh2BHKPzI8m+Yw0kAXNiiC5XMnjESMRX2DlA HClZqWQKXhjbdYrxmmSV2iLhS3Nrd6eIBeFkFlNbFPG4bLS99i2R24fNB496S1u4YdAn HRFj9MqKraWW05m9xyPAsrC8DM1VLyhE2/BL01mnYdOWkrdaM5urlywY29Aa+60yx8Qq Wf1dIOprZ3Cy2ENZcmtFrTgRA6EfsWeRBbKT590BhzWy9whFxLktTBVtEiHE+mAaW+Ig 6d2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=oAX6zCohmPiQ74kYACc2QgeImmemzleYDH4h+URdt/s=; b=E4QjSvVdq/v8N1ubr+7F4Gw8NnXwOzRxEHfOnlpPS4yHfRb+40j4ggByKFMbF2m8bj VvbU/Sxbb8kIsqzy/03HsA5lfNDZ0D+lXMH7aQ4sPHE9tUsy/yjhePV2Hbepaj2r9+UG v43fpoxMTLLMCfT2ralxTbLWJqaHKoceLRcuO2iaBu8+7uUZzPz89RieTyVBmk/pQsKd 0zonqlfcYh1D0FX4vffEoHtsOCH/01p5gd6WAAI0s7E1gUKIfyvwTq/tRFM7vxbVXaF7 wgmkn/gEjC8dOk0K1d2zw0TxWj+ZygoseZszWdeifU1EB3N///Nl17sWmHxmdW1rj3Ry U7+g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=o7Kj58FZ; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id l66si262015qkc.145.2019.04.29.20.09.03 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:09:03 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=o7Kj58FZ; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 0FA4D9A40; Mon, 29 Apr 2019 23:09:03 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:09:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=oAX6zCohmPiQ74kYACc2QgeImmemzleYDH4h+URdt/s=; b=o7Kj58FZ umNKgzxodgukfWTcut22REHE596NdqMjuUUUKvVQLx5BPJj0NTVZ3d36ynR8D3Zf hYyp6bB6IctA12jmf3nGBSK0wdsHhOvqQeEcd6b0LzA+naKDlxaYrR0fjB60JmUe 4MiUVO1e1yNCnRGW7kcjNv1oqoV8UyK75OUyGjJOA0g/FLXFgmnLTuDupzgBhM8P CUdo56GgKBC4qFvCVJ4hWuNHTAk2LScEPqiaSMNqyfTpyPm5X2L1/ufg5GGEaQnq YWqkh8PwrTQOaKGoBxk+JLF74PucyHy5GI4Mu/mGFFa507wP+nZKxpxaS6juK5kQ WUOXSWsMIge3rg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedv X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 12572103C8; Mon, 29 Apr 2019 23:08:54 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 03/15] slub: Sort slab cache list Date: Tue, 30 Apr 2019 13:07:34 +1000 Message-Id: <20190430030746.26102-4-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It is advantageous to have all defragmentable slabs together at the beginning of the list of slabs so that there is no need to scan the complete list. Put defragmentable caches first when adding a slab cache and others last. Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- mm/slab_common.c | 2 +- mm/slub.c | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/mm/slab_common.c b/mm/slab_common.c index 58251ba63e4a..db5e9a0b1535 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -393,7 +393,7 @@ static struct kmem_cache *create_cache(const char *name, goto out_free_cache; s->refcount = 1; - list_add(&s->list, &slab_caches); + list_add_tail(&s->list, &slab_caches); memcg_link_cache(s); out: if (err) diff --git a/mm/slub.c b/mm/slub.c index ae44d640b8c1..f6b0e4a395ef 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4342,6 +4342,8 @@ void kmem_cache_setup_mobility(struct kmem_cache *s, return; } + mutex_lock(&slab_mutex); + s->isolate = isolate; s->migrate = migrate; @@ -4350,6 +4352,10 @@ void kmem_cache_setup_mobility(struct kmem_cache *s, * to disable fast cmpxchg based processing. */ s->flags &= ~__CMPXCHG_DOUBLE; + + list_move(&s->list, &slab_caches); /* Move to top */ + + mutex_unlock(&slab_mutex); } EXPORT_SYMBOL(kmem_cache_setup_mobility); From patchwork Tue Apr 30 03:07:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922741 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E0FD41515 for ; Tue, 30 Apr 2019 03:09:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF0E328994 for ; Tue, 30 Apr 2019 03:09:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C0841289C9; Tue, 30 Apr 2019 03:09:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 26A1B28994 for ; Tue, 30 Apr 2019 03:09:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C3CF6B026C; Mon, 29 Apr 2019 23:09:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0747F6B026D; Mon, 29 Apr 2019 23:09:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E32306B026E; Mon, 29 Apr 2019 23:09:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id BEA586B026C for ; Mon, 29 Apr 2019 23:09:13 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id f82so928591qkb.9 for ; Mon, 29 Apr 2019 20:09:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=7KCR/bG3UyviFO0P3mTQEntswZ73LUL6n099eGv7Rtc=; b=Z8GQwJ2qpFxCQExlSvNOjMvPUUbZIw+hUC0WOKQBsxePRDjdzU4iICD42pxmdnoyQ6 ZrZcbiYo1Hvl64GNYTcCQzz8HwFNgBy6/RTzlxm4rHchewnxbPbed6T5a5dIriXrN1r1 9b8AP5oJ6grEskEZzDp0sF2zMfM6HdljOBlUys3rsDPQbIsD+RxewO8KgvorQRD104Bk 0s+NL2zbZBHBmFUogRpnqF38CfHda3wP2ULeZAcZAd34onD5D4cA5ZH8Q6PbFnXw9N6s 03hlmvRE/BlVr/UaUc2PYLjCCajOqucQFqBTdcQuzs0MFCxzYYEOhFIOWrj0/sfI49Km YynA== X-Gm-Message-State: APjAAAVn0Mh/lzyyCnd+NZ2rRN1qCGDPUCH6LsLrEw1DohHw/kN0qIRH XsJ+/nnK3ToPutT9F9T4zy4GtrEu8iuy5gpfWHA0dy6T5YPTBCi7mLLIG15gtMEgP5Z7sAB/YPT SQvtRXdHN2zhR/F9wdhHRHruXec7druop0q5uj2lCrfWyxdAV0o+DA1HG6rplr1Q= X-Received: by 2002:a0c:d0f0:: with SMTP id b45mr2507757qvh.138.1556593753492; Mon, 29 Apr 2019 20:09:13 -0700 (PDT) X-Google-Smtp-Source: APXvYqz/Pe+EKAS4NH8NMPvVVmdzX1wvFzluBrlRyB9I2MDSyGKCMkyvqSYRPzQdVonqvLHIjnlh X-Received: by 2002:a0c:d0f0:: with SMTP id b45mr2507679qvh.138.1556593752059; Mon, 29 Apr 2019 20:09:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593752; cv=none; d=google.com; s=arc-20160816; b=UgCiCp5HH5NLj/dTFXzx71VKyBPb7zwkxuJqyziVsvPLAhXK0SqXSzZc/DoB5fuqXg xvecJpFNXR9eWqmpOEl6BPrnf8iWiAwusYpKkWZeiKviGSPTB6Ll8aKNAxpPtm1sRn6A +c8z11088Hnp/d1Fv69aUXbc2/I1M0vWhkYEAXPsC6C3kaBv5n9mwI7Mej9ZhYgQdAsl bEMKsG9jKe2sf0VpIDQKEnbr7Js7u26rGuqKIciCkFYZhJ74ekX7WhrLGjm2ZxK9xCWw LeIlqVhX/6Tl+zTsi2pZ3uaHXhW9F89qPN/ldnzvaEr6aYegSZwcjQ35JxYcjyntTFqd eBrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=7KCR/bG3UyviFO0P3mTQEntswZ73LUL6n099eGv7Rtc=; b=0eLxlANelYHAqaSDQsmfjJyoS5cfMgfo06djiqqhzPr9NFjcQbYWHg4zkNA2nPENkE nGSObqnKK8mOhndn2zAmezrrV8cgrMtmo3mlR5Dj7iYbqUxPfiQ/Nr3mCLXjkqtAsmPJ RPScgjpW7z0SDhyob90XS0NCCY2vYHR3G2JS6DJPEV+iFV+S111VS3nzQoMK7A954//0 m3o5ch6bJyfR590ZeAqLTGBTSh4UOaQNwhh053tydGU4GOcCTeiH1jlhRe9ZZZPLRvy4 dMOD3uCZW+3nNPJsEguGCP8Qerv0AhX38DNjqP503c2wPuSwJweksodLdPMIf7dLyvem 6WnQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ZTLJLpXp; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id s48si11110302qte.335.2019.04.29.20.09.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:09:12 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ZTLJLpXp; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id C221F139E3; Mon, 29 Apr 2019 23:09:11 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:09:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=7KCR/bG3UyviFO0P3mTQEntswZ73LUL6n099eGv7Rtc=; b=ZTLJLpXp Fy4UgEvHxDVz6/DsgQSKUI6fxb63dJYyj6NCHYyjtod4LEBzdlOdWRgcD+zWZAQf rV7G4VmzRojZYJG4TQMbdtWNf6wKoEPsLchJ3fP6RScExbbu9a7YuvVzKWulhfb0 6VLKNLhDCk1QYmLiB9PCgq7SBOvaguKAH44uQv93SRNkA1HY7gqu5l/2Mxga89r+ GMIM0KonSi7iVqujKMiowbUT8Zv+Ca2jOrdvK7tAtendv3jM+3fv9eyQ7iMN8kZR tCHT7JNv/iFRZe2in9djJ8O3FX2WNN9Yb584jAzjtuZjTBz7X3H2n5+kbq+eZIOl Owzhvz6gNJlfQQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpeef X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id AD70D103CA; Mon, 29 Apr 2019 23:09:03 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 04/15] slub: Slab defrag core Date: Tue, 30 Apr 2019 13:07:35 +1000 Message-Id: <20190430030746.26102-5-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Internal fragmentation can occur within pages used by the slub allocator. Under some workloads large numbers of pages can be used by partial slab pages. This under-utilisation is bad simply because it wastes memory but also because if the system is under memory pressure higher order allocations may become difficult to satisfy. If we can defrag slab caches we can alleviate these problems. Implement Slab Movable Objects in order to defragment slab caches. Slab defragmentation may occur: 1. Unconditionally when __kmem_cache_shrink() is called on a slab cache by the kernel calling kmem_cache_shrink(). 2. Unconditionally through the use of the slabinfo command. slabinfo -s 3. Conditionally via the use of kmem_cache_defrag() - Use Slab Movable Objects when shrinking cache. Currently when the kernel calls kmem_cache_shrink() we curate the partial slabs list. If object migration is not enabled for the cache we still do this, if however, SMO is enabled we attempt to move objects in partially full slabs in order to defragment the cache. Shrink attempts to move all objects in order to reduce the cache to a single partial slab for each node. - Add conditional per node defrag via new function: kmem_defrag_slabs(int node). kmem_defrag_slabs() attempts to defragment all slab caches for node. Defragmentation is done conditionally dependent on MAX_PARTIAL _AND_ defrag_used_ratio. Caches are only considered for defragmentation if the number of partial slabs exceeds MAX_PARTIAL (per node). Also, defragmentation only occurs if the usage ratio of the slab is lower than the configured percentage (sysfs field added in this patch). Fragmentation ratios are measured by calculating the percentage of objects in use compared to the total number of objects that the slab page can accommodate. The scanning of slab caches is optimized because the defragmentable slabs come first on the list. Thus we can terminate scans on the first slab encountered that does not support defragmentation. kmem_defrag_slabs() takes a node parameter. This can either be -1 if defragmentation should be performed on all nodes, or a node number. Defragmentation may be disabled by setting defrag ratio to 0 echo 0 > /sys/kernel/slab//defrag_used_ratio - Add a defrag ratio sysfs field and set it to 30% by default. A limit of 30% specifies that more than 3 out of 10 available slots for objects need to be in use otherwise slab defragmentation will be attempted on the remaining objects. In order for a cache to be defragmentable the cache must support object migration (SMO). Enabling SMO for a cache is done via a call to the recently added function: void kmem_cache_setup_mobility(struct kmem_cache *, kmem_cache_isolate_func, kmem_cache_migrate_func); Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- Documentation/ABI/testing/sysfs-kernel-slab | 14 + include/linux/slab.h | 1 + include/linux/slub_def.h | 7 + mm/slub.c | 385 ++++++++++++++++---- 4 files changed, 334 insertions(+), 73 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-kernel-slab b/Documentation/ABI/testing/sysfs-kernel-slab index 29601d93a1c2..7770c03be6b4 100644 --- a/Documentation/ABI/testing/sysfs-kernel-slab +++ b/Documentation/ABI/testing/sysfs-kernel-slab @@ -180,6 +180,20 @@ Description: list. It can be written to clear the current count. Available when CONFIG_SLUB_STATS is enabled. +What: /sys/kernel/slab/cache/defrag_used_ratio +Date: February 2019 +KernelVersion: 5.0 +Contact: Christoph Lameter + Pekka Enberg , +Description: + The defrag_used_ratio file allows the control of how aggressive + slab fragmentation reduction works at reclaiming objects from + sparsely populated slabs. This is a percentage. If a slab has + less than this percentage of objects allocated then reclaim will + attempt to reclaim objects so that the whole slab page can be + freed. 0% specifies no reclaim attempt (defrag disabled), 100% + specifies attempt to reclaim all pages. The default is 30%. + What: /sys/kernel/slab/cache/deactivate_to_tail Date: February 2008 KernelVersion: 2.6.25 diff --git a/include/linux/slab.h b/include/linux/slab.h index 886fc130334d..4bf381b34829 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -149,6 +149,7 @@ struct kmem_cache *kmem_cache_create_usercopy(const char *name, void (*ctor)(void *)); void kmem_cache_destroy(struct kmem_cache *); int kmem_cache_shrink(struct kmem_cache *); +unsigned long kmem_defrag_slabs(int node); void memcg_create_kmem_cache(struct mem_cgroup *, struct kmem_cache *); void memcg_deactivate_kmem_caches(struct mem_cgroup *); diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index 2879a2f5f8eb..34c6f1250652 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -107,6 +107,13 @@ struct kmem_cache { unsigned int red_left_pad; /* Left redzone padding size */ const char *name; /* Name (only for display!) */ struct list_head list; /* List of slab caches */ + int defrag_used_ratio; /* + * Ratio used to check against the + * percentage of objects allocated in a + * slab page. If less than this ratio + * is allocated then reclaim attempts + * are made. + */ #ifdef CONFIG_SYSFS struct kobject kobj; /* For sysfs */ struct work_struct kobj_remove_work; diff --git a/mm/slub.c b/mm/slub.c index f6b0e4a395ef..e601c804ed79 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -354,6 +354,12 @@ static __always_inline void slab_lock(struct page *page) bit_spin_lock(PG_locked, &page->flags); } +static __always_inline int slab_trylock(struct page *page) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + return bit_spin_trylock(PG_locked, &page->flags); +} + static __always_inline void slab_unlock(struct page *page) { VM_BUG_ON_PAGE(PageTail(page), page); @@ -3643,6 +3649,7 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) set_cpu_partial(s); + s->defrag_used_ratio = 30; #ifdef CONFIG_NUMA s->remote_node_defrag_ratio = 1000; #endif @@ -3959,79 +3966,6 @@ void kfree(const void *x) } EXPORT_SYMBOL(kfree); -#define SHRINK_PROMOTE_MAX 32 - -/* - * kmem_cache_shrink discards empty slabs and promotes the slabs filled - * up most to the head of the partial lists. New allocations will then - * fill those up and thus they can be removed from the partial lists. - * - * The slabs with the least items are placed last. This results in them - * being allocated from last increasing the chance that the last objects - * are freed in them. - */ -int __kmem_cache_shrink(struct kmem_cache *s) -{ - int node; - int i; - struct kmem_cache_node *n; - struct page *page; - struct page *t; - struct list_head discard; - struct list_head promote[SHRINK_PROMOTE_MAX]; - unsigned long flags; - int ret = 0; - - flush_all(s); - for_each_kmem_cache_node(s, node, n) { - INIT_LIST_HEAD(&discard); - for (i = 0; i < SHRINK_PROMOTE_MAX; i++) - INIT_LIST_HEAD(promote + i); - - spin_lock_irqsave(&n->list_lock, flags); - - /* - * Build lists of slabs to discard or promote. - * - * Note that concurrent frees may occur while we hold the - * list_lock. page->inuse here is the upper limit. - */ - list_for_each_entry_safe(page, t, &n->partial, lru) { - int free = page->objects - page->inuse; - - /* Do not reread page->inuse */ - barrier(); - - /* We do not keep full slabs on the list */ - BUG_ON(free <= 0); - - if (free == page->objects) { - list_move(&page->lru, &discard); - n->nr_partial--; - } else if (free <= SHRINK_PROMOTE_MAX) - list_move(&page->lru, promote + free - 1); - } - - /* - * Promote the slabs filled up most to the head of the - * partial list. - */ - for (i = SHRINK_PROMOTE_MAX - 1; i >= 0; i--) - list_splice(promote + i, &n->partial); - - spin_unlock_irqrestore(&n->list_lock, flags); - - /* Release empty slabs */ - list_for_each_entry_safe(page, t, &discard, lru) - discard_slab(s, page); - - if (slabs_node(s, node)) - ret = 1; - } - - return ret; -} - #ifdef CONFIG_MEMCG static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) { @@ -4326,6 +4260,287 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags) return err; } +/* + * Allocate a slab scratch space that is sufficient to keep pointers to + * individual objects for all objects in cache and also a bitmap for the + * objects (used to mark which objects are active). + */ +static inline void *alloc_scratch(struct kmem_cache *s) +{ + unsigned int size = oo_objects(s->max); + + return kmalloc(size * sizeof(void *) + + BITS_TO_LONGS(size) * sizeof(unsigned long), + GFP_KERNEL); +} + +/* + * move_slab_page() - Move all objects in the given slab. + * @page: The slab we are working on. + * @scratch: Pointer to scratch space. + * @node: The target node to move objects to. + * + * If the target node is not the current node then the object is moved + * to the target node. If the target node is the current node then this + * is an effective way of defragmentation since the current slab page + * with its object is exempt from allocation. + */ +static void move_slab_page(struct page *page, void *scratch, int node) +{ + unsigned long objects; + struct kmem_cache *s; + unsigned long flags; + unsigned long *map; + void *private; + int count; + void *p; + void **vector = scratch; + void *addr = page_address(page); + + local_irq_save(flags); + slab_lock(page); + + BUG_ON(!PageSlab(page)); /* Must be a slab page */ + BUG_ON(!page->frozen); /* Slab must have been frozen earlier */ + + s = page->slab_cache; + objects = page->objects; + map = scratch + objects * sizeof(void **); + + /* Determine used objects */ + bitmap_fill(map, objects); + for (p = page->freelist; p; p = get_freepointer(s, p)) + __clear_bit(slab_index(p, s, addr), map); + + /* Build vector of pointers to objects */ + count = 0; + memset(vector, 0, objects * sizeof(void **)); + for_each_object(p, s, addr, objects) + if (test_bit(slab_index(p, s, addr), map)) + vector[count++] = p; + + if (s->isolate) + private = s->isolate(s, vector, count); + else + /* Objects do not need to be isolated */ + private = NULL; + + /* + * Pinned the objects. Now we can drop the slab lock. The slab + * is frozen so it cannot vanish from under us nor will + * allocations be performed on the slab. However, unlocking the + * slab will allow concurrent slab_frees to proceed. So the + * subsystem must have a way to tell from the content of the + * object that it was freed. + * + * If neither RCU nor ctor is being used then the object may be + * modified by the allocator after being freed which may disrupt + * the ability of the migrate function to tell if the object is + * free or not. + */ + slab_unlock(page); + local_irq_restore(flags); + + /* Perform callback to move the objects */ + s->migrate(s, vector, count, node, private); +} + +/* + * kmem_cache_defrag() - Defragment node. + * @s: cache we are working on. + * @node: The node to move objects from. + * @target_node: The node to move objects to. + * @ratio: The defrag ratio (percentage, between 0 and 100). + * + * Release slabs with zero objects and try to call the migration function + * for slabs with less than the 'ratio' percentage of objects allocated. + * + * Moved objects are allocated on @target_node. + * + * Return: The number of partial slabs left on @node after the + * operation. + */ +static unsigned long kmem_cache_defrag(struct kmem_cache *s, + int node, int target_node, int ratio) +{ + struct kmem_cache_node *n = get_node(s, node); + struct page *page, *page2; + LIST_HEAD(move_list); + unsigned long flags; + + if (node == target_node && n->nr_partial <= 1) { + /* + * Trying to reduce fragmentation on a node but there is + * only a single or no partial slab page. This is already + * the optimal object density that we can reach. + */ + return n->nr_partial; + } + + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &n->partial, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + if (page->inuse) { + if (page->inuse > ratio * page->objects / 100) { + slab_unlock(page); + /* + * Skip slab because the object density + * in the slab page is high enough. + */ + continue; + } + + list_move(&page->lru, &move_list); + if (s->migrate) { + /* Stop page being considered for allocations */ + n->nr_partial--; + page->frozen = 1; + } + slab_unlock(page); + } else { /* Empty slab page */ + list_del(&page->lru); + n->nr_partial--; + slab_unlock(page); + discard_slab(s, page); + } + } + + if (!s->migrate) { + /* + * No defrag method. By simply putting the zaplist at + * the end of the partial list we can let them simmer + * longer and thus increase the chance of all objects + * being reclaimed. + */ + list_splice(&move_list, n->partial.prev); + } + + spin_unlock_irqrestore(&n->list_lock, flags); + + if (s->migrate && !list_empty(&move_list)) { + void **scratch = alloc_scratch(s); + if (scratch) { + /* Try to remove / move the objects left */ + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + } + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + /* + * Objects left in slab page, move it to the + * tail of the partial list to increase the + * chance that the freeing of the remaining + * objects will free the slab page. + */ + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); + } + + return n->nr_partial; +} + +/** + * kmem_defrag_slabs() - Defrag slab caches. + * @node: The node to defrag or -1 for all nodes. + * + * Defrag slabs conditional on the amount of fragmentation in a page. + * + * Return: The total number of partial slabs in migratable caches left + * on @node after the operation. + */ +unsigned long kmem_defrag_slabs(int node) +{ + struct kmem_cache *s; + unsigned long left = 0; + int nid; + + if (node >= MAX_NUMNODES) + return -EINVAL; + + /* + * kmem_defrag_slabs() may be called from the reclaim path which + * may be called for any page allocator alloc. So there is the + * danger that we get called in a situation where slub already + * acquired the slub_lock for other purposes. + */ + if (!mutex_trylock(&slab_mutex)) + return 0; + + list_for_each_entry(s, &slab_caches, list) { + /* + * Defragmentable caches come first. If the slab cache is + * not defragmentable then we can stop traversing the list. + */ + if (!s->migrate) + break; + + if (node >= 0) { + if (s->node[node]->nr_partial > MAX_PARTIAL) { + left += kmem_cache_defrag(s, node, node, + s->defrag_used_ratio); + } + continue; + } + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (s->node[nid]->nr_partial > MAX_PARTIAL) { + left += kmem_cache_defrag(s, nid, nid, + s->defrag_used_ratio); + } + } + } + mutex_unlock(&slab_mutex); + return left; +} +EXPORT_SYMBOL(kmem_defrag_slabs); + +/** + * __kmem_cache_shrink() - Shrink a cache. + * @s: The cache to shrink. + * + * Reduces the memory footprint of a slab cache by as much as possible. + * + * This works by: + * 1. Removing empty slabs from the partial list. + * 2. Migrating slab objects to denser slab pages if the slab cache + * supports migration. If not, reorganizing the partial list so that + * more densely allocated slab pages come first. + * + * Not called directly, called by kmem_cache_shrink(). + */ +int __kmem_cache_shrink(struct kmem_cache *s) +{ + int node; + int left = 0; + + flush_all(s); + for_each_node_state(node, N_NORMAL_MEMORY) + left += kmem_cache_defrag(s, node, node, 100); + + return left; +} +EXPORT_SYMBOL(__kmem_cache_shrink); + void kmem_cache_setup_mobility(struct kmem_cache *s, kmem_cache_isolate_func isolate, kmem_cache_migrate_func migrate) @@ -5177,6 +5392,29 @@ static ssize_t destroy_by_rcu_show(struct kmem_cache *s, char *buf) } SLAB_ATTR_RO(destroy_by_rcu); +static ssize_t defrag_used_ratio_show(struct kmem_cache *s, char *buf) +{ + return sprintf(buf, "%d\n", s->defrag_used_ratio); +} + +static ssize_t defrag_used_ratio_store(struct kmem_cache *s, + const char *buf, size_t length) +{ + unsigned long ratio; + int err; + + err = kstrtoul(buf, 10, &ratio); + if (err) + return err; + + if (ratio > 100) + return -EINVAL; + + s->defrag_used_ratio = ratio; + return length; +} +SLAB_ATTR(defrag_used_ratio); + #ifdef CONFIG_SLUB_DEBUG static ssize_t slabs_show(struct kmem_cache *s, char *buf) { @@ -5501,6 +5739,7 @@ static struct attribute *slab_attrs[] = { &validate_attr.attr, &alloc_calls_attr.attr, &free_calls_attr.attr, + &defrag_used_ratio_attr.attr, #endif #ifdef CONFIG_ZONE_DMA &cache_dma_attr.attr, From patchwork Tue Apr 30 03:07:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922745 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B4F71515 for ; Tue, 30 Apr 2019 03:09:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1CF7828994 for ; Tue, 30 Apr 2019 03:09:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1052E289C9; Tue, 30 Apr 2019 03:09:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A311728994 for ; Tue, 30 Apr 2019 03:09:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A61BF6B026F; Mon, 29 Apr 2019 23:09:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A12816B0270; Mon, 29 Apr 2019 23:09:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 900106B0271; Mon, 29 Apr 2019 23:09:21 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id 705116B026F for ; Mon, 29 Apr 2019 23:09:21 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id n39so2776291qtn.0 for ; Mon, 29 Apr 2019 20:09:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=lcvJv7UGuJkFVkknd4VfvOV/45YfjtyDZP6dQPUSq5iCpWlJSahMrZD0UXHWsPYfBZ 7bE8nHoB4h2NWzEp04TovBQULwCr9qGcWiJ7VRgxyjsHyTjwmr9A6qGanoYAXB0qpqRA 4GB6bkRq7QzgCN5WW6cpbEvQ4lemMB3i+n41mXvz8wLYuSEFhHhs1Whoy2knKnTPmoR5 gWgr5AvC7c3zJKny8zfvr6BeyTtktDpyXhIjQT79u22ZvGCKlbrBLCUBE/XIBY+cznqp PnKnCIWyQbiELp2OARBw0iumBqSeo6xig5A74vpsPFYiC3R+HCwicW+8ZKSjOU2POukI jgtw== X-Gm-Message-State: APjAAAV3yFkK/nRpaQACIOlrPlkP7omFsefN9BGukhwCIi14JgKOSDsv F0oJCZj4nDp1f2Ry/Cr/ILEh6LPhcq7YCvEoXQzCIxym/mIASanxP4JwMBQkRBER+bFzfxv6S68 gmpdXtKwVrKt+fgG3/x5o8lqOWbxwZrvEeRqkdFjJ+0GcmYybfSxtswlTgAsmASU= X-Received: by 2002:a0c:a944:: with SMTP id z4mr36717760qva.119.1556593761245; Mon, 29 Apr 2019 20:09:21 -0700 (PDT) X-Google-Smtp-Source: APXvYqwfh7k+3f36oNPDdtHO08ixPShm+jf4YPRDXoiTole08whBbP3pFS5/Mp1BXB+neCC+B7o0 X-Received: by 2002:a0c:a944:: with SMTP id z4mr36717734qva.119.1556593760417; Mon, 29 Apr 2019 20:09:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593760; cv=none; d=google.com; s=arc-20160816; b=PjkwwNogL9+VNRCgQXkCZ0BuYphxv37Aa4/OMfn1kRQctpZFejFIseXNRj0u/0sBnT HYTh1H73hSGkGdcRYw1hQLNboBQPktqGr8w9sFYp5fOCVUPPgbYZJdRKlW4k87tJcWWX hh2bttY1LFCPEqIXGJHfxW2oDPu5VldkqNflb88afRpPMjcmxBRHzJ/PjJNHqAJJvzuP xPFWgxN/DjhscqApqJHFoTRl8VZHdioxoHTjr4AXADtHKJkwVBzW5DX++ZoWp4xZjdvD BywT9k3EWGSYvGoT4ogKzPHbLChwS9gBX7wnCBwciUtFkkEGxr7C6MWK1BIGuXyOiooQ Bxvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=1C0PRSWTmDUY5oMM97/PYi2rGxf6rgwnAtZpTB2QfkySA/uWtJnwtWpK74JjwrKUXc 4+GnVRE9UOdBia8VjDmHnLaYCgLTUUBBVFcq0+wudTQM8bU+k/42T66YGzggDYSvQo3o kuhap+bDDSgi4W6g3s9mxAEC9kfGnWfGCwZmxocI9AISL2s86Fss5iNuTc6eYNuzp46g 2VoEkl36byDlLmDyXWbBCNNO4Y+mVDcOEHfPI2aPd0y8i5MFCxk0kGbLESPG0P3PmuyX a/DWCDz7mxe4Bltz6qk7mUCgOohgl69X92cj6mmSzIrfwkpY85MtQ+aRbAK7yI3gucRE WJmQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="G/Duqu6R"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id g123si1833785qkd.172.2019.04.29.20.09.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:09:20 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b="G/Duqu6R"; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 269D8266A; Mon, 29 Apr 2019 23:09:20 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:09:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=rfKadsTflryLW7911DaQh925XQ0k2Bp3gi/G7eAxyJQ=; b=G/Duqu6R IX98r8fGsTHboSiPaGfc+oXz7F7GTHMM+xq5e9jY+x01fnRalMqbv3hYxJYD+uZp In3DEYyoYoQWtxbp7nwkNfxwIzekJplnUFlExEXLK+gw7KtIF585kzwLTEcCGZ++ 06i8aJw/HpRVmjOrjZ+zcRByTtIlESsm/K8avMuIcAqnKHMqEZZQepONBedgJ86m A7gBb+MrF+Uim8l1IKY6ZYJrysjVpKY0S1Q/GMa3HOeRPYOcXGEtHUUvCI4zAY0K wtxNew6HgyxMWft46wy4hw8mz0Tw8GlU5LZdVhYFzFoI9JUZYgGDk3KA0aqvocjE QNmgcFRnoU8vHg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpeeg X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 77B05103CB; Mon, 29 Apr 2019 23:09:12 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 05/15] tools/vm/slabinfo: Add remote node defrag ratio output Date: Tue, 30 Apr 2019 13:07:36 +1000 Message-Id: <20190430030746.26102-6-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add output line for NUMA remote node defrag ratio. Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index cbfc56c44c2f..d2c22f9ee2d8 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -34,6 +34,7 @@ struct slabinfo { unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; int movable, ctor; + int remote_node_defrag_ratio; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; unsigned long free_fastpath, free_slowpath; @@ -377,6 +378,10 @@ static void slab_numa(struct slabinfo *s, int mode) if (skip_zero && !s->slabs) return; + if (mode) { + printf("\nNUMA remote node defrag ratio: %3d\n", + s->remote_node_defrag_ratio); + } if (!line) { printf("\n%-21s:", mode ? "NUMA nodes" : "Slab"); for(node = 0; node <= highest_node; node++) @@ -1272,6 +1277,8 @@ static void read_slab_dir(void) slab->cpu_partial_free = get_obj("cpu_partial_free"); slab->alloc_node_mismatch = get_obj("alloc_node_mismatch"); slab->deactivate_bypass = get_obj("deactivate_bypass"); + slab->remote_node_defrag_ratio = + get_obj("remote_node_defrag_ratio"); chdir(".."); if (read_slab_obj(slab, "ops")) { if (strstr(buffer, "ctor :")) From patchwork Tue Apr 30 03:07:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922749 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AACED1515 for ; Tue, 30 Apr 2019 03:09:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A4DA28994 for ; Tue, 30 Apr 2019 03:09:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8D9B5289C9; Tue, 30 Apr 2019 03:09:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F6D528994 for ; Tue, 30 Apr 2019 03:09:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30E8F6B0272; Mon, 29 Apr 2019 23:09:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2EF516B0274; Mon, 29 Apr 2019 23:09:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1ACEB6B0275; Mon, 29 Apr 2019 23:09:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by kanga.kvack.org (Postfix) with ESMTP id ED8336B0272 for ; Mon, 29 Apr 2019 23:09:29 -0400 (EDT) Received: by mail-qt1-f197.google.com with SMTP id f48so12106058qtk.17 for ; Mon, 29 Apr 2019 20:09:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=oUffA13Wt4WX5wrfeeeLMbFXKS+cmRzVopuUttL5DTZ9xRQmBDh0oCZE3glYfphp30 huvyYDjHsvodyTm2aRk95vFzgddNHfvBHlJ0k4HEeNPeNSD8VqBWntvhsrBWSe2KG96C BwQ9bW6BvutrqA+K8z76OyPJPMJiZYEiohzKHlZtFZ7EOFK+o3ysoGGYZWt+DhfInRkA FkKBi6+iafPuDjc53KOF6vxhdFBegMLcv+Cu5W/fJgId7MywNST2py7ZWk0KM2ra2W59 XAdJ7VKJShVDdKZ9TgqDs/+woQLouVr8wuU8Gmhw6/v9QCMin7zlZh0+rd8j3crb2kH4 t43Q== X-Gm-Message-State: APjAAAV5dzqj3aUf7y7ZSq9TzwDlswvlgdkxNAC5S4p4rDpplceB3qQV v/P2dT0zENIZh5Atnf2TrocX7QQRpy5VMAhhaiCl8ShX1yDqPsUxXFmJtOGVzLreQ+7SaagrcB1 iqLQ7zuEUNdD8meko3iRq5VSuln0PuTC7SSQtgokFNCvI3adoJx5HLdF5Yf/HfEA= X-Received: by 2002:ac8:35c6:: with SMTP id l6mr41562588qtb.326.1556593769762; Mon, 29 Apr 2019 20:09:29 -0700 (PDT) X-Google-Smtp-Source: APXvYqxJ9ztFy5eUnaN8vzcesdRbH1fr/Hz3Uw0gssvajfGrSGwcmJzv+lIS54Al7DCp6TfsRNug X-Received: by 2002:ac8:35c6:: with SMTP id l6mr41562547qtb.326.1556593768826; Mon, 29 Apr 2019 20:09:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593768; cv=none; d=google.com; s=arc-20160816; b=FOEocoi6wY3M3Try+qYF/hi49kLCEkj/6F/IYM/XdBCdcUoO8qb58MGnSZU1pGryFF ZH5SOa66GafbaEgJHFbmXSL8NNORBAwKOmkieqnhU7I1fQXPwwVwtNW1Omk1PAt5M/07 +Q5y3LlEbHKong5D+s4udTiVXLzcC0S2l3vCC7uV/p8RHPFAPRzxwgbgyIQyoXS+FN4o JjpjxdcoHOsl5wxbFX5TVWtViXbwbD8FkL6DwWJR7q3Oa1DQzgblO0OdeALVISZIMVRj AhE4TgslmuPhyYZaCEKBAQwMpPBL1KgIg4Z2cpcPQM5mlReOWn93z15lNiltoez74bC3 WBdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=f22kMdSGwE4sEkB6F12O8+JM6Zy71FTvXMtWKZjKAnjIaFTnzL6R365XPx5N0uRXit sFR4QU4v0RubOBi1rvDcHi61+qiNp3e8dPQWgDRtfev/5m5qQvN7lFqoYX1R5AfbeLF3 tYaqiKOwoPoCXm+R1vMX1HgjyFVcTJoL4QyreqLz+v2bQH4yvMe76KSTx9JtNzmDxoBN EwqLC90y9phnMuzUrUF5CxMtVnHrRt7bao3BjVrjVCflH+YGhA7QsRf00pgZWC0mRMib wyc+RVlwhTf6TCKGfsIlNco2ZiF81dbFrLQSt7hWy4rOwJysBKuYOj5XsZdy5cL/EWhi AMlQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=IIiJcM6A; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id y66si1990782qke.252.2019.04.29.20.09.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:09:28 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=IIiJcM6A; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 8A2C4C248; Mon, 29 Apr 2019 23:09:28 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:09:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=eB8W4ZifzOOx7n477C9lsHSgteznueDWRYbTVC4RGG4=; b=IIiJcM6A Me6lXrEYm7aMvEu4IniFKE1Gy/ckecyeq+RQCjDcEluoWUY/TJPW6sh8UNZ+l3k2 nRzSrCQpCgS6c1MihLEx14Aqb0sDq3rzDhqQE2rBXsIqlEEX0OEFWdDk4O1+HlD1 Q5eciHxYnNKKJ+U/rB8aESqJSe+2vM6ttloEQdkdePxND+yaq286Gm0v++FZIVct lQkZNj3lpvoaWE4wBqSdc+heek7Rq7NxK36kScV3p6qxBzNBhde+US49CsQEQ+hI uKQkDPxz3P9ne2lS7AJxzKPqhaJqEab1e9Ou5FL6KhmkLKUWNQutfPyyUyplf60x zmjKkhoXnj0BSg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpeeh X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id C4DAA103CA; Mon, 29 Apr 2019 23:09:20 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 06/15] tools/vm/slabinfo: Add defrag_used_ratio output Date: Tue, 30 Apr 2019 13:07:37 +1000 Message-Id: <20190430030746.26102-7-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add output for the newly added defrag_used_ratio sysfs knob. Signed-off-by: Tobin C. Harding --- tools/vm/slabinfo.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c index d2c22f9ee2d8..ef4ff93df4cc 100644 --- a/tools/vm/slabinfo.c +++ b/tools/vm/slabinfo.c @@ -34,6 +34,7 @@ struct slabinfo { unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; int movable, ctor; + int defrag_used_ratio; int remote_node_defrag_ratio; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; @@ -549,6 +550,8 @@ static void report(struct slabinfo *s) printf("** Slabs are destroyed via RCU\n"); if (s->reclaim_account) printf("** Reclaim accounting active\n"); + if (s->movable) + printf("** Defragmentation at %d%%\n", s->defrag_used_ratio); printf("\nSizes (bytes) Slabs Debug Memory\n"); printf("------------------------------------------------------------------------\n"); @@ -1279,6 +1282,7 @@ static void read_slab_dir(void) slab->deactivate_bypass = get_obj("deactivate_bypass"); slab->remote_node_defrag_ratio = get_obj("remote_node_defrag_ratio"); + slab->defrag_used_ratio = get_obj("defrag_used_ratio"); chdir(".."); if (read_slab_obj(slab, "ops")) { if (strstr(buffer, "ctor :")) From patchwork Tue Apr 30 03:07:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922753 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E0C2C1390 for ; Tue, 30 Apr 2019 03:09:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD53F28994 for ; Tue, 30 Apr 2019 03:09:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C0B84289C9; Tue, 30 Apr 2019 03:09:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4BCC428994 for ; Tue, 30 Apr 2019 03:09:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24ADD6B0275; Mon, 29 Apr 2019 23:09:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 1FC096B0276; Mon, 29 Apr 2019 23:09:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C41C6B0277; Mon, 29 Apr 2019 23:09:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id D90AF6B0275 for ; Mon, 29 Apr 2019 23:09:39 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id c22so12160917qtk.10 for ; Mon, 29 Apr 2019 20:09:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=FywiKAR+ovXQUR3AQ/5KISp2k/uq2C+0OrYsI87IBWEOAHzjluPUXoM7Amw5g3UIOR AjnREwP3qTIvM85paV2GSCMev3WzoJV7AWAYaT8WE/2ryTeZyMW2JZG1dx7upKfyRRW+ FLywS8hB6VxZXhK1zTNb1DjWW818fKcG/mBYg4W4hxhx67aXBOpFL6RMTf4A75TV65r7 IoXF3gLFbsjalBfOIKk2S0JDcY+1Sd9Wc4aH1Rtvhp8hMthHMTTJGtaftEx51Y/fYBU0 79ofAssezNZ24qHdYd2dYnsC9yJpktcj9cQjTBL6Cj+YyR9V1otfa0pXXUz1CIS78wOA vCPw== X-Gm-Message-State: APjAAAX+TiOwYmqQUGxYKI3yvGozuhw9UQWaSUMyYx9h3OBYZ7iMrqFX D9Q+v4pAtj6XNxfPZHBLbc5dHCg7JB/SGa2SsBkUClSbuF9GLstQAHXn6bzsa5Ikq2zcKCOW39z dHZYdkZYC48j2K1VFZy+jEpLdBhlHbIvhGtDhSrGEqvL733J47jZ0f0VSKsUVscs= X-Received: by 2002:ac8:2831:: with SMTP id 46mr52335248qtq.366.1556593779587; Mon, 29 Apr 2019 20:09:39 -0700 (PDT) X-Google-Smtp-Source: APXvYqwNh+Z4GeiaGVhlnQ+zVcqSTd6LyHFnf3hbelV0/SwQ+TFN2cQXNYbodlVOjM7y1/0Nbx0m X-Received: by 2002:ac8:2831:: with SMTP id 46mr52335163qtq.366.1556593777541; Mon, 29 Apr 2019 20:09:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593777; cv=none; d=google.com; s=arc-20160816; b=DUfACW5wHg5qBNh9YtTfkkPcwjWuUGqLJFYZQnJymcdJcUMIPj26NEt4pvb4lmby3b 7pwGDmkyfQJBLql/OR1Enm7rUB556LaJWX0C9e8TeRgswHGk1e5OuTFHduX+qee3dk0s GJVpyC00lhDKSUW0o0EVdRE2ePIwZydaVDt4nFr2011Wn9hT/ClelX+3IGfvm+u5hCuq Yg8Q0jb7q9MR9+z+kgDUych6Jla0yYalaBwLrI4OXFPhOf6aX6IES256hn3HCaBNCYwH Kz2EcoDyWdyAHnfU+Pj9P6UilKW/pX/PPJoBkZY6yGR6wwjhQB1MblgaUw7hbhWs4JGF YN7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=DizsKmexKnfk4GyGxW8aEYZ6AGHIrCYDhLHJiN5yHSKbcpvURZ8YhNFSknLkQ+qp8Z tnjINIt7yQGilwDLHTAbCCoFy14usuvaK69CWSOdJ4pDp2dmzWFnl/FOBoPbIFpLwwaN 52cHoNGhQd+oJJzBGDtzwcvU5zsxzpdpuRBNVEfiiKJZFEZaJ+DzcZSREgzV59xfnm7D 5gvkBH6lb3E4cVZmvBv1xTqzn4fc+mOg5wOjxgttDmvgdQTpXgyLito9Ra3PxssM9o5H DWtlMtmV1o6GqxSfXpNh2G1mSgEBbvl//Qfl95DNzWFVD17a/8hqc2IwaB6SzSDWvzRg AKNQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=u429WcMU; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id w5si12057219qkj.56.2019.04.29.20.09.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:09:37 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=u429WcMU; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 33AD06758; Mon, 29 Apr 2019 23:09:37 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:09:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=r+9QuCXW7/td+AqWATFlgqfuY4wjDJcwrAnptYZPsCY=; b=u429WcMU mXINl+wHr1/9u31BzBnEWPLbjPYZU8JXztnGxSlrBhAwBVx78Ws8ynJIs+ImPem4 1cv0daSGYBST6C6UkgwoBMOm7Ll8kKfbg6itM+GEBSv5Qz5DXiSR0J9Jc9dl3kWE Dmvx5kQXR+Hz9ndLdD4XMz9Qe9E+tEs6RZzHsIHt5nP2PQ1T0aDzu0Fwhy4XiGEv 8xpYVhLfnIyYYvUiwfyjTJz4RTUIvRPHq1NouOlH//phVQFk7QjgCGMxRGsVtvuK yaewJlPRajmm+966r/NtMy31n4zJPEoE5NEgNhA0pEufEXiXgIoNbyGFqI3tmck2 7IA/VS6evxG45A== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpeei X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 24E7C103CA; Mon, 29 Apr 2019 23:09:28 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 07/15] tools/testing/slab: Add object migration test module Date: Tue, 30 Apr 2019 13:07:38 +1000 Message-Id: <20190430030746.26102-8-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just implemented slab movable objects for the SLUB allocator. We should test that code. In order to do so we need to be able to do a number of things - Create a cache - Enable Slab Movable Objects for the cache - Allocate objects to the cache - Free objects from within specific slabs of the cache We can do all this via a loadable module. Add a module that defines functions that can be triggered from userspace via a debugfs entry. From the source: /* * SLUB defragmentation a.k.a. Slab Movable Objects (SMO). * * This module is used for testing the SLUB allocator. Enables * userspace to run kernel functions via a debugfs file. * * debugfs: /sys/kernel/debugfs/smo/callfn (write only) * * String written to `callfn` is parsed by the module and associated * function is called. See fn_tab for mapping of strings to functions. */ References to allocated objects are kept by the module in a linked list so that userspace can control which object to free. We introduce the following four functions via the function table "enable": Enables object migration for the test cache. "alloc X": Allocates X objects "free X [Y]": Frees X objects starting at list position Y (default Y==0) "test": Runs [stress] tests from within the module (see below). {"enable", smo_enable_cache_mobility}, {"alloc", smo_alloc_objects}, {"free", smo_free_object}, {"test", smo_run_module_tests}, Freeing from the start of the list creates a hole in the slab being freed from (i.e. creates a partial slab). The results of running these commands can be see using `slabinfo` (available in tools/vm/): make -o slabinfo tools/vm/slabinfo.c Stress tests can be run from within the module. These tests are internal to the module because we verify that object references are still good after object migration. These are called 'stress' tests because it is intended that they create/free a lot of objects. Userspace can control the number of objects to create, default is 1000. Example test session -------------------- Relevant /proc/slabinfo column headers: name # mount -t debugfs none /sys/kernel/debug/ $ cd path/to/linux/tools/testing/slab; make ... # insmod slub_defrag.ko # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 0 0 392 20 2 From this we can see that the module created cache 'smo_test' with 20 objects per slab and 2 pages per slab (and cache is currently empty). We can play with the slab allocator manually: # insmod slub_defrag.ko # echo 'alloc 21' > callfn # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 21 40 392 20 2 We see here that 21 active objects have been allocated creating 2 slabs (40 total objects). # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 21 Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 2 Sanity Checks : On Total: 16384 SlabObj: 392 Full : 1 Redzoning : On Used : 1176 SlabSiz: 8192 Partial: 1 Poisoning : On Loss : 15208 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 7056 Align : 8 Objects: 20 Tracing : Off Lpadd: 704 Now free an object from the first slot of the first slab # echo 'free 1' > callfn # cat /proc/slabinfo | grep smo_test | sed 's/:.*//' smo_test 20 40 392 20 2 # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 20 Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 2 Sanity Checks : On Total: 16384 SlabObj: 392 Full : 0 Redzoning : On Used : 1120 SlabSiz: 8192 Partial: 2 Poisoning : On Loss : 15264 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 6720 Align : 8 Objects: 20 Tracing : Off Lpadd: 704 Calling shrink now on the cache does nothing because object migration is not enabled (output omitted). If we enable object migration then shrink the cache we expect the object from the second slab to me moved to the first slot in the first slab and the second slab to be removed from the partial list. # echo 'enable' > callfn # slabinfo smo_test --shrink # slabinfo smo_test --report Slabcache: smo_test Aliases: 0 Order : 1 Objects: 20 ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 56 Total : 1 Sanity Checks : On Total: 8192 SlabObj: 392 Full : 1 Redzoning : On Used : 1120 SlabSiz: 8192 Partial: 0 Poisoning : On Loss : 7072 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 6720 Align : 8 Objects: 20 Tracing : Off Lpadd: 352 We can run the stress tests (with the default number of objects): # cd /sys/kernel/debug/smo # echo 'test' > callfn [ 3.576617] smo: test using nr_objs: 1000 keep: 10 [ 3.580169] smo: Module tests completed successfully Signed-off-by: Tobin C. Harding --- tools/testing/slab/Makefile | 10 + tools/testing/slab/slub_defrag.c | 566 +++++++++++++++++++++++++++++++ 2 files changed, 576 insertions(+) create mode 100644 tools/testing/slab/Makefile create mode 100644 tools/testing/slab/slub_defrag.c diff --git a/tools/testing/slab/Makefile b/tools/testing/slab/Makefile new file mode 100644 index 000000000000..440c2e3e356f --- /dev/null +++ b/tools/testing/slab/Makefile @@ -0,0 +1,10 @@ +obj-m += slub_defrag.o + +KTREE=../../.. + +all: + make -C ${KTREE} M=$(PWD) modules + +clean: + make -C ${KTREE} M=$(PWD) clean + diff --git a/tools/testing/slab/slub_defrag.c b/tools/testing/slab/slub_defrag.c new file mode 100644 index 000000000000..4a5c24394b96 --- /dev/null +++ b/tools/testing/slab/slub_defrag.c @@ -0,0 +1,566 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +/* + * SLUB defragmentation a.k.a. Slab Movable Objects (SMO). + * + * This module is used for testing the SLUB allocator. Enables + * userspace to run kernel functions via a debugfs file. + * + * debugfs: /sys/kernel/debugfs/smo/callfn (write only) + * + * String written to `callfn` is parsed by the module and associated + * function is called. See fn_tab for mapping of strings to functions. + */ + +/* debugfs commands accept two optional arguments */ +#define SMO_CMD_DEFAUT_ARG -1 + +#define SMO_DEBUGFS_DIR "smo" +struct dentry *smo_debugfs_root; + +#define SMO_CACHE_NAME "smo_test" +static struct kmem_cache *cachep; + +struct smo_slub_object { + struct list_head list; + char buf[32]; /* Unused except to control size of object */ + long id; +}; + +/* Our list of allocated objects */ +LIST_HEAD(objects); + +static void list_add_to_objects(struct smo_slub_object *so) +{ + /* + * We free from the front of the list so store at the + * tail in order to put holes in the cache when we free. + */ + list_add_tail(&so->list, &objects); +} + +/** + * smo_object_ctor() - SMO object constructor function. + * @ptr: Pointer to memory where the object should be constructed. + */ +void smo_object_ctor(void *ptr) +{ + struct smo_slub_object *so = ptr; + + INIT_LIST_HEAD(&so->list); + memset(so->buf, 0, sizeof(so->buf)); + so->id = -1; +} + +/** + * smo_cache_migrate() - kmem_cache migrate function. + * @cp: kmem_cache pointer. + * @objs: Array of pointers to objects to migrate. + * @size: Number of objects in @objs. + * @node: NUMA node where the object should be allocated. + * @private: Pointer returned by kmem_cache_isolate_func(). + */ +void smo_cache_migrate(struct kmem_cache *cp, void **objs, int size, + int node, void *private) +{ + struct smo_slub_object **so_objs = (struct smo_slub_object **)objs; + struct smo_slub_object *so_old, *so_new; + int i; + + for (i = 0; i < size; i++) { + so_old = so_objs[i]; + + so_new = kmem_cache_alloc_node(cachep, GFP_KERNEL, node); + if (!so_new) { + pr_debug("kmem_cache_alloc failed\n"); + return; + } + + /* Copy object */ + so_new->id = so_old->id; + + /* Update references to old object */ + list_del(&so_old->list); + list_add_to_objects(so_new); + + kmem_cache_free(cachep, so_old); + } +} + +static int smo_enable_cache_mobility(int _unused, int __unused) +{ + /* Enable movable objects: BOOM! */ + kmem_cache_setup_mobility(cachep, NULL, smo_cache_migrate); + pr_info("smo: kmem_cache %s defrag enabled\n", SMO_CACHE_NAME); + return 0; +} + +/* + * smo_alloc_objects() - Allocate objects and store reference. + * @nr_objs: Number of objects to allocate. + * @node: NUMA node to allocate objects on. + * + * Allocates @n smo_slub_objects. Stores a reference to them in + * the global list of objects (at the tail of the list). + * + * Return: The number of objects allocated. + */ +static int smo_alloc_objects(int nr_objs, int node) +{ + struct smo_slub_object *so; + int i; + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1; + if (node == SMO_CMD_DEFAUT_ARG) + node = NUMA_NO_NODE; + + for (i = 0; i < nr_objs; i++) { + if (node == NUMA_NO_NODE) + so = kmem_cache_alloc(cachep, GFP_KERNEL); + else + so = kmem_cache_alloc_node(cachep, GFP_KERNEL, node); + if (!so) { + pr_err("smo: Failed to alloc object %d of %d\n", i, nr_objs); + return i; + } + list_add_to_objects(so); + } + return nr_objs; +} + +/* + * smo_free_object() - Frees n objects from position. + * @nr_objs: Number of objects to free. + * @pos: Position in global list to start freeing. + * + * Iterates over the global list of objects to position @pos then frees @n + * objects from there (or to end of list). Does nothing if @n > list length. + * + * Calling with @n==0 frees all objects starting at @pos. + * + * Return: Number of objects freed. + */ +static int smo_free_object(int nr_objs, int pos) +{ + struct smo_slub_object *cur, *tmp; + int deleted = 0; + int i = 0; + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1; + if (pos == SMO_CMD_DEFAUT_ARG) + pos = 0; + + list_for_each_entry_safe(cur, tmp, &objects, list) { + if (i < pos) { + i++; + continue; + } + + list_del(&cur->list); + kmem_cache_free(cachep, cur); + deleted++; + if (deleted == nr_objs) + break; + } + return deleted; +} + +static int index_for_expected_id(long *expected, int size, long id) +{ + int i; + + /* Array is unsorted, just iterate the whole thing */ + for (i = 0; i < size; i++) { + if (expected[i] == id) + return i; + } + return -1; /* Not found */ +} + +static int assert_have_objects(int nr_objs, int keep) +{ + struct smo_slub_object *cur; + long *expected; /* Array of expected IDs */ + int nr_ids; /* Length of array */ + long id; + int index, i; + + nr_ids = nr_objs / keep + 1; + + expected = kmalloc_array(nr_ids, sizeof(long), GFP_KERNEL); + if (!expected) + return -ENOMEM; + + id = 0; + for (i = 0; i < nr_ids; i++) { + expected[i] = id; + id += keep; + } + + list_for_each_entry(cur, &objects, list) { + index = index_for_expected_id(expected, nr_ids, cur->id); + if (index < 0) { + pr_err("smo: ID not found: %ld\n", cur->id); + return -1; + } + + if (expected[index] == -1) { + pr_err("smo: ID already encountered: %ld\n", cur->id); + return -1; + } + expected[index] = -1; + } + return 0; +} + +/* + * smo_run_module_tests() - Runs unit tests from within the module + * @nr_objs: Number of objects to allocate. + * @keep: Free all but 1 in @keep objects. + * + * Allocates @nr_objects then iterates over the allocated objects + * freeing all but 1 out of every @keep objects i.e. for @keep==10 + * keeps the first object then frees the next 9. + * + * Caller is responsible for ensuring that the cache has at most a + * single slab on the partial list without any objects in it. This is + * easy enough to ensure, just call this when the module is freshly + * loaded. + */ +static int smo_run_module_tests(int nr_objs, int keep) +{ + struct smo_slub_object *so; + struct smo_slub_object *cur, *tmp; + long i; + + if (!list_empty(&objects)) { + pr_err("smo: test requires clean module state\n"); + return -1; + } + + /* Set sane parameters if no args passed in */ + if (nr_objs == SMO_CMD_DEFAUT_ARG) + nr_objs = 1000; + if (keep == SMO_CMD_DEFAUT_ARG) + keep = 10; + + pr_info("smo: test using nr_objs: %d keep: %d\n", nr_objs, keep); + + /* Perhaps we got called like this 'test 1000' */ + if (keep == 0) { + pr_err("Usage: test \n"); + return -1; + } + + /* Test constructor */ + so = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!so) { + pr_err("smo: Failed to alloc object\n"); + return -1; + } + if (so->id != -1) { + pr_err("smo: Initial state incorrect"); + return -1; + } + kmem_cache_free(cachep, so); + + /* + * Test that object migration is correctly implemented by module + * + * This gives us confidence that if new code correctly enables + * object migration (via correct implementation of migrate and + * isolate functions) then the slub allocator code that does + * object migration is correct. + */ + + for (i = 0; i < nr_objs; i++) { + so = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!so) { + pr_err("smo: Failed to alloc object %ld of %d\n", + i, nr_objs); + return -1; + } + so->id = (long)i; + list_add_to_objects(so); + } + + assert_have_objects(nr_objs, 1); + + i = 0; + list_for_each_entry_safe(cur, tmp, &objects, list) { + if (i++ % keep == 0) + continue; + + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + + /* Verify shrink does nothing when migration is not enabled */ + kmem_cache_shrink(cachep); + assert_have_objects(nr_objs, 1); + + /* Now test shrink */ + kmem_cache_setup_mobility(cachep, NULL, smo_cache_migrate); + kmem_cache_shrink(cachep); + /* + * Because of how migrate function deletes and adds objects to + * the objects list we have no way of knowing the order. We + * want to confirm that we have all the objects after shrink + * that we had before we did the shrink. + */ + assert_have_objects(nr_objs, keep); + + /* cleanup */ + list_for_each_entry_safe(cur, tmp, &objects, list) { + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + kmem_cache_shrink(cachep); /* Remove empty slabs from partial list */ + + pr_info("smo: Module tests completed successfully\n"); + return 0; +} + +/* + * struct functions() - Map command to a function pointer. + */ +struct functions { + char *fn_name; + int (*fn_ptr)(int arg0, int arg1); +} fn_tab[] = { + /* + * Because of the way we parse the function table no command + * may have another command as its prefix. + * i.e. this will break: 'foo' and 'foobar' + */ + {"enable", smo_enable_cache_mobility}, + {"alloc", smo_alloc_objects}, + {"free", smo_free_object}, + {"test", smo_run_module_tests}, +}; + +#define FN_TAB_SIZE (sizeof(fn_tab) / sizeof(struct functions)) + +/* + * parse_cmd_buf() - Gets command and arguments command string. + * @buf: Buffer containing the command string. + * @cmd: Out parameter, pointer to the command. + * @arg1: Out parameter, stores the first argument. + * @arg2: Out parameter, stores the second argument. + * + * Parses and tokenizes the input command buffer. Stores a pointer to the + * command (start of @buf) in @cmd. Stores the converted long values for + * argument 1 and 2 in the respective out parameters @arg1 and @arg2. + * + * Since arguments are optional, if they are not found the default values are + * returned. In order for the caller to differentiate defaults from arguments + * of the same value the number of arguments parsed is returned. + * + * Return: Number of arguments found. + */ +static int parse_cmd_buf(char *buf, char **cmd, long *arg1, long *arg2) +{ + int found; + char *ptr; + int ret; + + *arg1 = SMO_CMD_DEFAUT_ARG; + *arg2 = SMO_CMD_DEFAUT_ARG; + found = 0; + + /* Jump over the command, check if there are any args */ + ptr = strsep(&buf, " "); + if (!ptr || !buf) + return found; + + ptr = strsep(&buf, " "); + ret = kstrtol(ptr, 10, arg1); + if (ret < 0) { + pr_err("failed to convert arg, defaulting to %d. (%s)\n", + SMO_CMD_DEFAUT_ARG, ptr); + return found; + } + found++; + if (!buf) /* No second arg */ + return found; + + ptr = strsep(&buf, " "); + ret = kstrtol(ptr, 10, arg2); + if (ret < 0) { + pr_err("failed to convert arg, defaulting to %d. (%s)\n", + SMO_CMD_DEFAUT_ARG, ptr); + return found; + } + found++; + + return found; +} + +/* + * call_function() - Calls the function described by str. + * @str: ' []' + * + * Does table lookup on , calls appropriate function passing + * as a the argument. Optional arg defaults to 1. + */ +static void call_function(char *str) +{ + char *cmd; + long arg1 = 0; + long arg2 = 0; + int i; + + if (!str) + return; + + (void)parse_cmd_buf(str, &cmd, &arg1, &arg2); + + for (i = 0; i < FN_TAB_SIZE; i++) { + char *fn_name = fn_tab[i].fn_name; + + if (strcmp(fn_name, str) == 0) { + fn_tab[i].fn_ptr(arg1, arg2); + return; /* All done */ + } + } + + pr_err("failed to call function for cmd: %s\n", str); +} + +/* + * smo_callfn_debugfs_write() - debugfs write function. + * @file: User file + * @user_buf: Userspace buffer + * @len: Length of the user space buffer + * @off: Offset within the file + * + * Used for triggering functions by writing command to debugfs file. + * + * echo ' ' > /sys/kernel/debug/smo/callfn + * + * Return: Number of bytes copied if request succeeds, + * the corresponding error code otherwise. + */ +static ssize_t smo_callfn_debugfs_write(struct file *file, + const char __user *ubuf, + size_t len, + loff_t *off) +{ + char *kbuf; + int nbytes = 0; + + if (*off != 0 || len == 0) + return -EINVAL; + + kbuf = kzalloc(len, GFP_KERNEL); + if (!kbuf) + return -ENOMEM; + + nbytes = strncpy_from_user(kbuf, ubuf, len); + if (nbytes < 0) + goto out; + + if (kbuf[nbytes - 1] == '\n') + kbuf[nbytes - 1] = '\0'; + + call_function(kbuf); /* Tokenizes kbuf */ +out: + kfree(kbuf); + return nbytes; +} + +const struct file_operations fops_callfn_debugfs = { + .owner = THIS_MODULE, + .write = smo_callfn_debugfs_write, +}; + +static int __init smo_debugfs_init(void) +{ + struct dentry *d; + + smo_debugfs_root = debugfs_create_dir(SMO_DEBUGFS_DIR, NULL); + d = debugfs_create_file("callfn", 0200, smo_debugfs_root, NULL, + &fops_callfn_debugfs); + if (IS_ERR(d)) + return PTR_ERR(d); + + return 0; +} + +static void __exit smo_debugfs_cleanup(void) +{ + debugfs_remove_recursive(smo_debugfs_root); +} + +static int __init smo_cache_init(void) +{ + cachep = kmem_cache_create(SMO_CACHE_NAME, + sizeof(struct smo_slub_object), + 0, 0, smo_object_ctor); + if (!cachep) + return -1; + + return 0; +} + +static void __exit smo_cache_cleanup(void) +{ + struct smo_slub_object *cur, *tmp; + + list_for_each_entry_safe(cur, tmp, &objects, list) { + list_del(&cur->list); + kmem_cache_free(cachep, cur); + } + kmem_cache_destroy(cachep); +} + +static int __init smo_init(void) +{ + int ret; + + ret = smo_cache_init(); + if (ret) { + pr_err("smo: Failed to create cache\n"); + return ret; + } + pr_info("smo: Created kmem_cache: %s\n", SMO_CACHE_NAME); + + ret = smo_debugfs_init(); + if (ret) { + pr_err("smo: Failed to init debugfs\n"); + return ret; + } + pr_info("smo: Created debugfs directory: /sys/kernel/debugfs/%s\n", + SMO_DEBUGFS_DIR); + + pr_info("smo: Test module loaded\n"); + return 0; +} +module_init(smo_init); + +static void __exit smo_exit(void) +{ + smo_debugfs_cleanup(); + smo_cache_cleanup(); + + pr_info("smo: Test module removed\n"); +} +module_exit(smo_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Tobin C. Harding"); +MODULE_DESCRIPTION("SLUB Movable Objects test module."); From patchwork Tue Apr 30 03:07:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922757 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DE1981515 for ; Tue, 30 Apr 2019 03:09:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC96028994 for ; Tue, 30 Apr 2019 03:09:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C078D289C9; Tue, 30 Apr 2019 03:09:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 99D2B28994 for ; Tue, 30 Apr 2019 03:09:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7F00A6B000A; Mon, 29 Apr 2019 23:09:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7A08F6B0278; Mon, 29 Apr 2019 23:09:48 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 61ADD6B0279; Mon, 29 Apr 2019 23:09:48 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id 425AA6B000A for ; Mon, 29 Apr 2019 23:09:48 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id 144so10658995qkf.12 for ; Mon, 29 Apr 2019 20:09:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=TNKDdc3S1GoziLR7/byPvML87T5VizlolAZNpQQ1RVO1g3+OxRgI59uns8yeMclZ5i Mg2nnwMHUvuQBWUT8vIexQjohlWC/56KJfnk07buBxuY6+aTjtDPahrOGJJ3tpsUm4Hp cbUmwmCcrX7Z5ps9I4MtYekguV2rN9P2pUy6vIE4CXdd7e6a0AVDC4YNF0eRUI5GXvH8 hPy0DiZdP4l0ef+L9SkfgD7tN4YWvXq+Ctg4hnnazolCgQOvu1IgolZ9kLrPPYs5YsVt gqdEQpuCrlhFO2Iu1zFynHPi+zOAdYdklJvnbOBACWjOdkKxkz2tFt7kkfTmUyV5JY3/ Qppg== X-Gm-Message-State: APjAAAUS2unanL/m8d1rSf1k6aZD5L2etLd6J0WQ1Yq4QZxG7RKwURb1 OfY7Y6Lp7fLO8ucBmyvS4XU9e8igvYO0I7cl4b6oU2Yy3d8apA9k9Y4WHKk/gmw0B6VE/T45XFI XxKMZ+xpWPI6Dac0YAikeJXCXWj/7BqrYArtW5uSO1T9BCNWlhmvmr7K5GnKa5qc= X-Received: by 2002:a0c:9568:: with SMTP id m37mr19956231qvm.154.1556593787979; Mon, 29 Apr 2019 20:09:47 -0700 (PDT) X-Google-Smtp-Source: APXvYqyLJLNAJ+ydnnKyb+gDwQLATVCudhFw+2sT0+f8R4N2t1mzNNUIjieArFSIoLQ2ZQmO70vi X-Received: by 2002:a0c:9568:: with SMTP id m37mr19956147qvm.154.1556593785862; Mon, 29 Apr 2019 20:09:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593785; cv=none; d=google.com; s=arc-20160816; b=mH/pYW2gly+9DMegvDKxSyrxhP2+bSsdALXXDcpdhd60SyBwdbh28bnhAFOCqk732r 5fq+IkYPYeD04lVWjs4BuSrfSzgKyDGU5TBeZUYAG/1tQDLcwKGmzVEyrj3Co/nHsyUM W72OmOeo7MV5bQr3w5Ho/6uNgmBh1ozLpf/QUDPe6gaS7PJGxxo+Ss+OVHgKhhPuIQ4a adByf5bRiZ8t0TE2LPT/5zoOgwyXXibi8OlCyBAB5adgaBCfMPpF5Xvkk2edY0SVGgEt ROz4Wo+hzapC8iEqVH3mtR3vbbyrNBUg+pwtzcGxVKRaCXuO1GGHhROR7pwLo2sBzGBl dWFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=RkZDFYYugKL5x937s9pUWOgvgdH2JNajxI2pP4O/O47SFVZn0uO8S99XkHw59Pi7J5 nF4rJ3zLhh+hetsRHtDsGQaTCba4OiubUWBCzjKGrVnrCOQy9meFAtbtkNrsgXHEzbSe LAvc8FgNKYzPu15Yp+4vB8SnPKU8NV0u4r1W26vziLpOP5g/bivdiXvRqqtv+0aAP4TA 2vOGGNaYjjfvhJdMe5pXTJYCs1gZDV7lS2BESLZx9Tnq8AKKgyaVhxPE+0f1NIDcMfID tb9Z40zmlpJdbqXflZD7G30AiExiOn1LmyRq4gFHK3/qM9d7uZn2+v0izoNIjU66tI9P K1mg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ASlhx7fp; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id c3si7006878qvu.149.2019.04.29.20.09.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:09:45 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ASlhx7fp; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 903B91392C; Mon, 29 Apr 2019 23:09:45 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:09:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=uFjBqY0LQHef1Kn1ZeUrrNjMW5b2u14kh3M2b0Ph/vA=; b=ASlhx7fp vSJyjBUzrvLn0cirv+vvTDmjhZccvtcSlfRFDcSnSHeGjKCI0S5lr7omxFjSVUrV 4ZagTm3higSZaQlUmOBeUW6A5IdlC6mNtE2nthdgDgX2z7vuQAUnfdaWzVa874Zg 8F26WyEnu500/au2vfpVoKsTREDEQ//GF9nRaop4DR1mY8OUeAl0ACppZui0cona sgQYqeLW6GPjNFxDmmMA99becdpkfldHTOPOtnl7SD28nA/jHoic/ygyu1zopTWy 19538xVI6g1qUEw4B/pBZaS6LP2HHYQSce/ifdOC9VbjliguMoSoOPsH6KOWfz+p zSw0cz9c0D1t1g== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpeej X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id B98CF103C9; Mon, 29 Apr 2019 23:09:37 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 08/15] tools/testing/slab: Add object migration test suite Date: Tue, 30 Apr 2019 13:07:39 +1000 Message-Id: <20190430030746.26102-9-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just added a module that enables testing the SLUB allocators ability to defrag/shrink caches via movable objects. Tests are better when they are automated. Add automated testing via a python script for SLUB movable objects. Example output: $ cd path/to/linux/tools/testing/slab $ /slub_defrag.py Please run script as root $ sudo ./slub_defrag.py $ sudo ./slub_defrag.py --debug Loading module ... Slab cache smo_test created Objects per slab: 20 Running sanity checks ... Running module stress test (see dmesg for additional test output) ... Removing module slub_defrag ... Loading module ... Slab cache smo_test created Running test non-movable ... testing slab 'smo_test' prior to enabling movable objects ... verified non-movable slabs are NOT shrinkable Running test movable ... testing slab 'smo_test' after enabling movable objects ... verified movable slabs are shrinkable Removing module slub_defrag ... Signed-off-by: Tobin C. Harding --- tools/testing/slab/slub_defrag.c | 1 + tools/testing/slab/slub_defrag.py | 451 ++++++++++++++++++++++++++++++ 2 files changed, 452 insertions(+) create mode 100755 tools/testing/slab/slub_defrag.py diff --git a/tools/testing/slab/slub_defrag.c b/tools/testing/slab/slub_defrag.c index 4a5c24394b96..8332e69ee868 100644 --- a/tools/testing/slab/slub_defrag.c +++ b/tools/testing/slab/slub_defrag.c @@ -337,6 +337,7 @@ static int smo_run_module_tests(int nr_objs, int keep) /* * struct functions() - Map command to a function pointer. + * If you update this please update the documentation in slub_defrag.py */ struct functions { char *fn_name; diff --git a/tools/testing/slab/slub_defrag.py b/tools/testing/slab/slub_defrag.py new file mode 100755 index 000000000000..41747c0db39b --- /dev/null +++ b/tools/testing/slab/slub_defrag.py @@ -0,0 +1,451 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 + +import subprocess +import sys +from os import path + +# SLUB Movable Objects test suite. +# +# Requirements: +# - CONFIG_SLUB=y +# - CONFIG_SLUB_DEBUG=y +# - The slub_defrag module in this directory. + +# Test SMO using a kernel module that enables triggering arbitrary +# kernel code from userspace via a debugfs file. +# +# Module code is in ./slub_defrag.c, basically the functionality is as +# follows: +# +# - Creates debugfs file /sys/kernel/debugfs/smo/callfn +# - Writes to 'callfn' are parsed as a command string and the function +# associated with command is called. +# - Defines 4 commands (all commands operate on smo_test cache): +# - 'test': Runs module stress tests. +# - 'alloc N': Allocates N slub objects +# - 'free N POS': Frees N objects starting at POS (see below) +# - 'enable': Enables SLUB Movable Objects +# +# The module maintains a list of allocated objects. Allocation adds +# objects to the tail of the list. Free'ing frees from the head of the +# list. This has the effect of creating free slots in the slab. For +# finer grained control over where in the cache slots are free'd POS +# (position) argument may be used. + +# The main() function is reasonably readable; the test suite does the +# following: +# +# 1. Runs the module stress tests. +# 2. Tests the cache without movable objects enabled. +# - Creates multiple partial slabs as explained above. +# - Verifies that partial slabs are _not_ removed by shrink (see below). +# 3. Tests the cache with movable objects enabled. +# - Creates multiple partial slabs as explained above. +# - Verifies that partial slabs _are_ removed by shrink (see below). + +# The sysfs file /sys/kernel/slab//shrink enables calling the +# function kmem_cache_shrink() (see mm/slab_common.c and mm/slub.cc). +# Shrinking a cache attempts to consolidate all partial slabs by moving +# objects if object migration is enable for the cache, otherwise +# shrinking a cache simply re-orders the partial list so as most densely +# populated slab are at the head of the list. + +# Enable/disable debugging output (also enabled via -d | --debug). +debug = False + +# Used in debug messages and when running `insmod`. +MODULE_NAME = "slub_defrag" + +# Slab cache created by the test module. +CACHE_NAME = "smo_test" + +# Set by get_slab_config() +objects_per_slab = 0 +pages_per_slab = 0 +debugfs_mounted = False # Set to true if we mount debugfs. + + +def eprint(*args, **kwargs): + print(*args, file=sys.stderr, **kwargs) + + +def dprint(*args, **kwargs): + if debug: + print(*args, file=sys.stderr, **kwargs) + + +def run_shell(cmd): + return subprocess.call([cmd], shell=True) + + +def run_shell_get_stdout(cmd): + return subprocess.check_output([cmd], shell=True) + + +def assert_root(): + user = run_shell_get_stdout('whoami') + if user != b'root\n': + eprint("Please run script as root") + sys.exit(1) + + +def mount_debugfs(): + mounted = False + + # Check if debugfs is mounted at a known mount point. + ret = run_shell('mount -l | grep /sys/kernel/debug > /dev/null 2>&1') + if ret != 0: + run_shell('mount -t debugfs none /sys/kernel/debug/') + mounted = True + dprint("Mounted debugfs on /sys/kernel/debug") + + return mounted + + +def umount_debugfs(): + dprint("Un-mounting debugfs") + run_shell('umount /sys/kernel/debug') + + +def load_module(): + """Loads the test module. + + We need a clean slab state to start with so module must + be loaded by the test suite. + """ + ret = run_shell('lsmod | grep %s > /dev/null' % MODULE_NAME) + if ret == 0: + eprint("Please unload slub_defrag module before running test suite") + return -1 + + dprint('Loading module ...') + ret = run_shell('insmod %s.ko' % MODULE_NAME) + if ret != 0: # ret==1 on error + return -1 + + dprint("Slab cache %s created" % CACHE_NAME) + return 0 + + +def unload_module(): + ret = run_shell('lsmod | grep %s > /dev/null' % MODULE_NAME) + if ret == 0: + dprint('Removing module %s ...' % MODULE_NAME) + run_shell('rmmod %s > /dev/null 2>&1' % MODULE_NAME) + + +def get_sysfs_value(filename): + """ + Parse slab sysfs files (single line: '20 N0=20') + """ + path = '/sys/kernel/slab/smo_test/%s' % filename + f = open(path, "r") + s = f.readline() + tokens = s.split(" ") + + return int(tokens[0]) + + +def get_nr_objects_active(): + return get_sysfs_value('objects') + + +def get_nr_objects_total(): + return get_sysfs_value('total_objects') + + +def get_nr_slabs_total(): + return get_sysfs_value('slabs') + + +def get_nr_slabs_partial(): + return get_sysfs_value('partial') + + +def get_nr_slabs_full(): + return get_nr_slabs_total() - get_nr_slabs_partial() + + +def get_slab_config(): + """Get relevant information from sysfs.""" + global objects_per_slab + + objects_per_slab = get_sysfs_value('objs_per_slab') + if objects_per_slab < 0: + return -1 + + dprint("Objects per slab: %d" % objects_per_slab) + return 0 + + +def verify_state(nr_objects_active, nr_objects_total, + nr_slabs_partial, nr_slabs_full, nr_slabs_total, msg=''): + err = 0 + got_nr_objects_active = get_nr_objects_active() + got_nr_objects_total = get_nr_objects_total() + got_nr_slabs_partial = get_nr_slabs_partial() + got_nr_slabs_full = get_nr_slabs_full() + got_nr_slabs_total = get_nr_slabs_total() + + if got_nr_objects_active != nr_objects_active: + err = -1 + + if got_nr_objects_total != nr_objects_total: + err = -2 + + if got_nr_slabs_partial != nr_slabs_partial: + err = -3 + + if got_nr_slabs_full != nr_slabs_full: + err = -4 + + if got_nr_slabs_total != nr_slabs_total: + err = -5 + + if err != 0: + dprint("Verify state: %s" % msg) + dprint(" what\t\t\twant\tgot") + dprint("-----------------------------------------") + dprint(" %s\t%d\t%d" % ('nr_objects_active', nr_objects_active, got_nr_objects_active)) + dprint(" %s\t%d\t%d" % ('nr_objects_total', nr_objects_total, got_nr_objects_total)) + dprint(" %s\t%d\t%d" % ('nr_slabs_partial', nr_slabs_partial, got_nr_slabs_partial)) + dprint(" %s\t\t%d\t%d" % ('nr_slabs_full', nr_slabs_full, got_nr_slabs_full)) + dprint(" %s\t%d\t%d\n" % ('nr_slabs_total', nr_slabs_total, got_nr_slabs_total)) + + return err + + +def exec_via_sysfs(command): + ret = run_shell('echo %s > /sys/kernel/debug/smo/callfn' % command) + if ret != 0: + eprint("Failed to echo command to sysfs: %s" % command) + + return ret + + +def enable_movable_objects(): + return exec_via_sysfs('enable') + + +def alloc(n): + exec_via_sysfs("alloc %d" % n) + + +def free(n, pos = 0): + exec_via_sysfs('free %d %d' % (n, pos)) + + +def shrink(): + ret = run_shell('slabinfo smo_test -s') + if ret != 0: + eprint("Failed to execute slabinfo -s") + + +def sanity_checks(): + # Verify everything is 0 to start with. + return verify_state(0, 0, 0, 0, 0, "sanity check") + + +def test_non_movable(): + one_over = objects_per_slab + 1 + + dprint("testing slab 'smo_test' prior to enabling movable objects ...") + + alloc(one_over) + + objects_active = one_over + objects_total = objects_per_slab * 2 + slabs_partial = 1 + slabs_full = 1 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: initial allocation") + if ret != 0: + eprint("test_non_movable: failed to verify initial state") + return -1 + + # Free object from first slot of first slab. + free(1) + objects_active = one_over - 1 + objects_total = objects_per_slab * 2 + slabs_partial = 2 + slabs_full = 0 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: after free") + if ret != 0: + eprint("test_non_movable: failed to verify after free") + return -1 + + # Non-movable cache, shrink should have no effect. + shrink() + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "non-movable: after shrink") + if ret != 0: + eprint("test_non_movable: failed to verify after shrink") + return -1 + + # Cleanup + free(objects_per_slab) + shrink() + + dprint("verified non-movable slabs are NOT shrinkable") + return 0 + + +def test_movable(): + one_over = objects_per_slab + 1 + + dprint("testing slab 'smo_test' after enabling movable objects ...") + + alloc(one_over) + + objects_active = one_over + objects_total = objects_per_slab * 2 + slabs_partial = 1 + slabs_full = 1 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: initial allocation") + if ret != 0: + eprint("test_movable: failed to verify initial state") + return -1 + + # Free object from first slot of first slab. + free(1) + objects_active = one_over - 1 + objects_total = objects_per_slab * 2 + slabs_partial = 2 + slabs_full = 0 + slabs_total = 2 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: after free") + if ret != 0: + eprint("test_movable: failed to verify after free") + return -1 + + # movable cache, shrink should move objects and free slab. + shrink() + objects_active = one_over - 1 + objects_total = objects_per_slab * 1 + slabs_partial = 0 + slabs_full = 1 + slabs_total = 1 + ret = verify_state(objects_active, objects_total, + slabs_partial, slabs_full, slabs_total, + "movable: after shrink") + if ret != 0: + eprint("test_movable: failed to verify after shrink") + return -1 + + # Cleanup + free(objects_per_slab) + shrink() + + dprint("verified movable slabs are shrinkable") + return 0 + + +def dprint_start_test(test): + dprint("Running %s ..." % test) + + +def dprint_done(): + dprint("") + + +def run_test(fn, desc): + dprint_start_test(desc) + ret = fn() + if ret < 0: + fail_test(desc) + dprint_done() + + +# Load and unload the module for this test to ensure clean state. +def run_module_stress_test(): + dprint("Running module stress test (see dmesg for additional test output) ...") + + unload_module() + ret = load_module() + if ret < 0: + cleanup_and_exit(ret) + + exec_via_sysfs("test"); + + unload_module() + + dprint() + + +def fail_test(msg): + eprint("\nFAIL: test failed: '%s' ... aborting\n" % msg) + cleanup_and_exit(1) + + +def display_help(): + print("Usage: %s [OPTIONS]\n" % path.basename(sys.argv[0])) + print("\tRuns defrag test suite (a.k.a. SLUB Movable Objects)\n") + print("OPTIONS:") + print("\t-d | --debug Enable verbose debug output") + print("\t-h | --help Print this help and exit") + + +def cleanup_and_exit(return_code): + global debugfs_mounted + + if debugfs_mounted == True: + umount_debugfs() + + unload_module() + + sys.exit(return_code) + + +def main(): + global debug + + if len(sys.argv) > 1: + if sys.argv[1] == '-h' or sys.argv[1] == '--help': + display_help() + sys.exit(0) + + if sys.argv[1] == '-d' or sys.argv[1] == '--debug': + debug = True + + assert_root() + + # Use cleanup_and_exit() instead of sys.exit() after mounting debugfs. + debugfs_mounted = mount_debugfs() + + # Loads and unloads the module. + run_module_stress_test() + + ret = load_module() + if (ret < 0): + cleanup_and_exit(ret) + + ret = get_slab_config() + if (ret != 0): + fail_test("get slab config details") + + run_test(sanity_checks, "sanity checks") + + run_test(test_non_movable, "test non-movable") + + ret = enable_movable_objects() + if (ret != 0): + fail_test("enable movable objects") + + run_test(test_movable, "test movable") + + cleanup_and_exit(0) + +if __name__== "__main__": + main() From patchwork Tue Apr 30 03:07:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922761 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 50D601390 for ; Tue, 30 Apr 2019 03:09:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 40F6F2861E for ; Tue, 30 Apr 2019 03:09:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 345CD2868C; Tue, 30 Apr 2019 03:09:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A853F2861E for ; Tue, 30 Apr 2019 03:09:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A12AE6B0279; Mon, 29 Apr 2019 23:09:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9C3F16B027B; Mon, 29 Apr 2019 23:09:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 88A4E6B027A; Mon, 29 Apr 2019 23:09:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 684416B000E for ; Mon, 29 Apr 2019 23:09:55 -0400 (EDT) Received: by mail-qt1-f199.google.com with SMTP id n1so12115701qte.12 for ; Mon, 29 Apr 2019 20:09:55 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=+np6ZQiGNtF1RzthBJCokkLQhVYLo5zEKFMlgFr5PgA=; b=sWJH3aup+d7Av7rDi6TnXZ3pVNjPPj0nzu0vu8VIzPbRFvLt/0HsRnj6KsBj+UsQhM N1k+6auaGui3JzU6aqvqIErm9lqE3S8QRgu0FNhhYm9LZrMx2ekVLuiNTtYYxJe1uZkt +1WW/W1eeAG9j8jGs0ZGZF22bdD3uEYvfi7KDokB6UKn0Lno5wEo5dhvhTMVtJ0fjC1T 03rVGwhqg16CianM//tHb2ADW5To7SfCQ8TuRGgSdXax3lp+gfb+ofZwhVD3fWNfETCz nXhv2Fif1PGp9JcDfbkaZkeJRI6lvONaEfWm/qL3ChtSrkr9sqCjjdmNqrFhMj/IiBP0 uDGg== X-Gm-Message-State: APjAAAXjWyeaRs7Rk1sf+kpBtZW+zW915fklZ8FtJ6JFTmfbnPK4swSF 2NFScuAwvk07a14pMxMPRbkuiGXoTixkSBTVGkIfxfS+csUHfvm33jcT6MA84WpcMeuyeXD+7wZ uPcLrvSCDkfTk4xfulF2y6g7N42K8So/m3wRdzoQnValiqlDZA5OFwMn04bgFRok= X-Received: by 2002:a05:620a:135b:: with SMTP id c27mr31384747qkl.92.1556593795162; Mon, 29 Apr 2019 20:09:55 -0700 (PDT) X-Google-Smtp-Source: APXvYqxYb9QrfoSC9hRoCdXs9iTAgfTtnqqLoUK1zMMMeIF7p1O419ZLnY+Y21zsZrxXVnCroZTC X-Received: by 2002:a05:620a:135b:: with SMTP id c27mr31384704qkl.92.1556593794095; Mon, 29 Apr 2019 20:09:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593794; cv=none; d=google.com; s=arc-20160816; b=t2p9AP0UOcltJI8C4k3bsx4c+vmeSRgqSkhqmt7iWqw+CHoFJdUw/AJ81mM8PzyB+8 E/4knbhfq+0UXkkub3glrY3erlfjstNG9VUY40B3YBvgigDnRG+HBOn811rmfLQC9cYw eHxZZ8jczZLcI45ch9TEpL94RTMXyR+hsv0LJo0ZKK+ZpZq3U6FCtZ1G+Xazd1Of0BQz MODqal9JMoHSWnekWx6jvOuHh8mCJ/+rcA1wCEsrz9iAygcBfaDJnDTNhTaYgG71BY+G AJZk/Le4vQTNNjFIw1HiPbANAxYdhqfpqGmzLRMvpOeqElbK3RijNvIm0IhvjqRn1C9A Lw1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=+np6ZQiGNtF1RzthBJCokkLQhVYLo5zEKFMlgFr5PgA=; b=XciiySl1O0e913N668bpJGawoJpxXnWx+nBrCPhEGLzhoRU1B61vvc9jD1Qv08OIhM e5odOl8RDEmW2gKLaOxisPYv+t3NMFeZkPVYpTHw20Q8TuGhg5waqUEXm/xsiPcjPaE8 BsmQ8RithD+qYaomzJHAWW9w2Ua3JQ9DbY928DaZefnLkSYfOhjBSUafeq6FxfZSqB5/ HWwl2m3/kS5lUMWhcLeqL6Y14vMww+Kkwb2Z2BDq+hPNTZe1/scsEdzwWjDvQ+4RmFSB eFOcvM01Os6LBX/tIPz0t6x3s9SM3vht4NcnDUIzjzqFPobAkuUOTp5RzYyxA842XBeM 3T+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=O4HjqKHF; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id 47si1144761qtn.4.2019.04.29.20.09.54 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:09:54 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=O4HjqKHF; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id C2BE713DC3; Mon, 29 Apr 2019 23:09:53 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:09:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=+np6ZQiGNtF1RzthBJCokkLQhVYLo5zEKFMlgFr5PgA=; b=O4HjqKHF BMRO+50OXaEG3RE7O+zTcbPMuVoq49Tf/5PF9HsXBPI5e79VDS2G7Btpm7streiL mPYGpSb5EqoTGXMbit0ASRHdTO/VuEdFnkGXxKQJvpP9FdpAX9qc3uQieAXr4LC+ EviJ7DdeG/qj2woFW13fjvIrWGhgpkRH1fFGlOn522Ie9ktUS2M63bbpixQkBlrr A6P8SddlC74ucI3tn1Vowqk8H9ltLaS5a5voileP8jc2CKMakD3pnO2QRMpoZHDJ HOWtOLCIYUkb7I5yLLZs0KwfoXKe8YhWXrRDTRMBkS2ormWiEgNN1BaO1TNEM7fd WlL7efT8rA9huQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpeek X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 118F3103CB; Mon, 29 Apr 2019 23:09:45 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 09/15] xarray: Implement migration function for objects Date: Tue, 30 Apr 2019 13:07:40 +1000 Message-Id: <20190430030746.26102-10-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Implement functions to migrate objects. This is based on initial code by Matthew Wilcox and was modified to work with slab object migration. This patch can not be merged until all radix tree & IDR users are converted to the XArray because xa_nodes and radix tree nodes share the same slab cache (thanks Matthew). Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- lib/radix-tree.c | 13 +++++++++++++ lib/xarray.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 62 insertions(+) diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 14d51548bea6..9412c2853726 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -1613,6 +1613,17 @@ static int radix_tree_cpu_dead(unsigned int cpu) return 0; } +extern void xa_object_migrate(void *tree_node, int numa_node); + +static void radix_tree_migrate(struct kmem_cache *s, void **objects, int nr, + int node, void *private) +{ + int i; + + for (i = 0; i < nr; i++) + xa_object_migrate(objects[i], node); +} + void __init radix_tree_init(void) { int ret; @@ -1627,4 +1638,6 @@ void __init radix_tree_init(void) ret = cpuhp_setup_state_nocalls(CPUHP_RADIX_DEAD, "lib/radix:dead", NULL, radix_tree_cpu_dead); WARN_ON(ret < 0); + kmem_cache_setup_mobility(radix_tree_node_cachep, NULL, + radix_tree_migrate); } diff --git a/lib/xarray.c b/lib/xarray.c index 6be3acbb861f..731dd3d8ddb8 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1971,6 +1971,55 @@ void xa_destroy(struct xarray *xa) } EXPORT_SYMBOL(xa_destroy); +void xa_object_migrate(struct xa_node *node, int numa_node) +{ + struct xarray *xa = READ_ONCE(node->array); + void __rcu **slot; + struct xa_node *new_node; + int i; + + /* Freed or not yet in tree then skip */ + if (!xa || xa == XA_RCU_FREE) + return; + + new_node = kmem_cache_alloc_node(radix_tree_node_cachep, + GFP_KERNEL, numa_node); + if (!new_node) + return; + + xa_lock_irq(xa); + + /* Check again..... */ + if (xa != node->array) { + node = new_node; + goto unlock; + } + + memcpy(new_node, node, sizeof(struct xa_node)); + + if (list_empty(&node->private_list)) + INIT_LIST_HEAD(&new_node->private_list); + else + list_replace(&node->private_list, &new_node->private_list); + + for (i = 0; i < XA_CHUNK_SIZE; i++) { + void *x = xa_entry_locked(xa, new_node, i); + + if (xa_is_node(x)) + rcu_assign_pointer(xa_to_node(x)->parent, new_node); + } + if (!new_node->parent) + slot = &xa->xa_head; + else + slot = &xa_parent_locked(xa, new_node)->slots[new_node->offset]; + rcu_assign_pointer(*slot, xa_mk_node(new_node)); + +unlock: + xa_unlock_irq(xa); + xa_node_free(node); + rcu_barrier(); +} + #ifdef XA_DEBUG void xa_dump_node(const struct xa_node *node) { From patchwork Tue Apr 30 03:07:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922765 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 512CF1390 for ; Tue, 30 Apr 2019 03:10:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3FBBC28646 for ; Tue, 30 Apr 2019 03:10:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 335E128681; Tue, 30 Apr 2019 03:10:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E503728646 for ; Tue, 30 Apr 2019 03:10:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA66A6B0269; Mon, 29 Apr 2019 23:10:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id E55EE6B027C; Mon, 29 Apr 2019 23:10:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D1E9F6B027D; Mon, 29 Apr 2019 23:10:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id B106F6B0269 for ; Mon, 29 Apr 2019 23:10:04 -0400 (EDT) Received: by mail-qt1-f198.google.com with SMTP id n1so12115938qte.12 for ; Mon, 29 Apr 2019 20:10:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=VuD415CRnkHNDiY3468gW5h1W5mvBZlGhijm1uvXElFU+7pdge6s2EOqGpWoeggWKN B8xbSONZTzoDghluVHeVj7w+D4ZVyjEGdHsBc9LtLirAAKII99gR1Kxd2wrdOX1/KAen irqD5PDN5hkWvFt7z9UVgFQMsZLfbVgJWsGx6CnHtHsEB4WtiOAqEFii2zQReOen9gFn 9WvJL0etHe7v1TPmKfQx2KYo6tRNkSY3MACkLMfdiyi1T7bSmZGxyZI0H9DhEkBAfYZ/ OGujnzhk9B9KLIft9h02qDoirnwnS90pLX2fX1eGxj99qPkcYiAR35oPSB6msoSF4bIz HxSw== X-Gm-Message-State: APjAAAXJDLdCgJf+yaPPZncNuH8LssXif8OEkf5GsRQq+EcQeteuuxtH zoOKm21ixIu14MIgKIwSTfdZH0KScWfiEhuM+yJzrVEBUF90KCi42he42m/UZXzlAGowDzVokwy RAzCvQoHyUhI4Y4cMihkqvT5a2DRSyDxcUv722SjuJBkb8eyJYupcqAoV0Oqss7o= X-Received: by 2002:a0c:b29b:: with SMTP id r27mr44992272qve.241.1556593804451; Mon, 29 Apr 2019 20:10:04 -0700 (PDT) X-Google-Smtp-Source: APXvYqzeQQK2w2TtYDL8iq/YsOTkjkLJTGUwOGbqHoIAXVoFt1YUb0jrdCuoiYefpvG/tCcvWdq4 X-Received: by 2002:a0c:b29b:: with SMTP id r27mr44992193qve.241.1556593802963; Mon, 29 Apr 2019 20:10:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593802; cv=none; d=google.com; s=arc-20160816; b=w56TgSOb5msHaNTuviLhxOcYnbusRhWmjmwzyjpz/2wZhvPS4afucVAuqwahLnp8Yh jUeIVhoT+RDtVWy8OS8XKoz6K5XD19x0nAdFimwq7zXoX6FxVesH2avkMADBQxRG4MB+ Prqw1/qtNHN0fAiTqSmDDTvKkESTyAOIZ0dihwwziT9f9Rs1i2OGcISy5kjmpaVGURSU Dr9Z/3b1VPiPqKKhv8ajNYxXgnWUY6To6EzXLX776p3lt5HWbZyBM8bW8iMV9NnXrzui TWx5WbvfdibkXUwnOYp8+KYFJSMizh5A/vaT05G1LcKK5CLzeufkLL5Z5pfCRxFHm6jj +0Yw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=cjeX67yrsv1XnKbA730AhdGkpULpFqra3Im7CYcpXNV8eSyfhF6JEKro9EeEFuKFHa xuWPp/OUlBSN+tmSSBYEQX2WHBdnwWWkfosRbyxggbDDGzTlK5eGbvtmmTMEMlhmAKy9 M/V/2Ufd0udGNZGA0z4IVeMRftddk2wRoPRYJn2j6u+wyE7D0/ODDQBrE2lAZXpaNtCI 9cn5vRmraAgkKlC9j4jOJLmISi4NrTa6F5MkLTEuZvxcG3zu+8rdiZ8GOWmBWTOYIgy6 jTIpMBkWEcqFoRT6I7dPYLlDug4EAgE10FYHrpEiexTbU8ZXYw1TV0artRbB2zwRduCr u5bA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=XTK+XwH0; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id h51si2640383qth.340.2019.04.29.20.10.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:10:02 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=XTK+XwH0; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 9B88C5286; Mon, 29 Apr 2019 23:10:02 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:10:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=K23AG6kPeiLizgBxJZXoFVpxoNhX/Z2JwBtP24yd/wE=; b=XTK+XwH0 oxEqp6vnwjK6+caFT9VVtKjWU9dPddmaa+FjY/fKul/+C3ozeHUt+PTz67oo+YFp dO0KXmSWxeH3Gn8ZMuihSs0FzKtgc6OtxwHW9tDjfIKWzMe/5zlF7CCi3JlUZ8/m cmmzjPi7BSdf1RWoEv8Ve8GpBdWt1muMdyO7n3chIMm53Hc2iYNQuNQ+0Jnoi1y9 i1dH8Th5Qsben43b0KwnrQPDdx0xf7yLDBzCR+a3TRr14wuIFdeZsvf0qd6kjpoj uiV7WNISxy6CEB2sd8DIprq5su2zF0tt/fO4oKTdPhYDv3NUZDR1crmcUinqQ89Q 2MqixREsWub7Fw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpeel X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 6213F103CB; Mon, 29 Apr 2019 23:09:54 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 10/15] tools/testing/slab: Add XArray movable objects tests Date: Tue, 30 Apr 2019 13:07:41 +1000 Message-Id: <20190430030746.26102-11-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We just implemented movable objects for the XArray. Let's test it intree. Add test module for the XArray's movable objects implementation. Functionality of the XArray Slab Movable Object implementation can usually be seen by simply by using `slabinfo` on a running machine since the radix tree is typically in use on a running machine and will have partial slabs. For repeated testing we can use the test module to run to simulate a workload on the XArray then use `slabinfo` to test object migration is functioning. If testing on freshly spun up VM (low radix tree workload) it may be necessary to load/unload the module a number of times to create partial slabs. Example test session -------------------- Relevant /proc/slabinfo column headers: name Prior to testing slabinfo report for radix_tree_node: # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8352 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 497 Sanity Checks : On Total: 8142848 SlabObj: 912 Full : 473 Redzoning : On Used : 4810752 SlabSiz: 16384 Partial: 24 Poisoning : On Loss : 3332096 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2806272 Align : 8 Objects: 17 Tracing : Off Lpadd: 437360 Here you can see the kernel was built with Slab Movable Objects enabled for the XArray (XArray uses the radix tree below the surface). After inserting the test module (note we have triggered allocation of a number of radix tree nodes increasing the object count but decreasing the number of partial slabs): # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8442 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 499 Sanity Checks : On Total: 8175616 SlabObj: 912 Full : 484 Redzoning : On Used : 4862592 SlabSiz: 16384 Partial: 15 Poisoning : On Loss : 3313024 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2836512 Align : 8 Objects: 17 Tracing : Off Lpadd: 439120 Now we can shrink the radix_tree_node cache: # slabinfo radix_tree_node --shrink # slabinfo radix_tree_node --report Slabcache: radix_tree_node Aliases: 0 Order : 2 Objects: 8515 ** Reclaim accounting active ** Defragmentation at 30% Sizes (bytes) Slabs Debug Memory ------------------------------------------------------------------------ Object : 576 Total : 501 Sanity Checks : On Total: 8208384 SlabObj: 912 Full : 500 Redzoning : On Used : 4904640 SlabSiz: 16384 Partial: 1 Poisoning : On Loss : 3303744 Loss : 336 CpuSlab: 0 Tracking : On Lalig: 2861040 Align : 8 Objects: 17 Tracing : Off Lpadd: 440880 Note the single remaining partial slab. Signed-off-by: Tobin C. Harding --- tools/testing/slab/Makefile | 2 +- tools/testing/slab/slub_defrag_xarray.c | 211 ++++++++++++++++++++++++ 2 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 tools/testing/slab/slub_defrag_xarray.c diff --git a/tools/testing/slab/Makefile b/tools/testing/slab/Makefile index 440c2e3e356f..44c18d9a4d52 100644 --- a/tools/testing/slab/Makefile +++ b/tools/testing/slab/Makefile @@ -1,4 +1,4 @@ -obj-m += slub_defrag.o +obj-m += slub_defrag.o slub_defrag_xarray.o KTREE=../../.. diff --git a/tools/testing/slab/slub_defrag_xarray.c b/tools/testing/slab/slub_defrag_xarray.c new file mode 100644 index 000000000000..41143f73256c --- /dev/null +++ b/tools/testing/slab/slub_defrag_xarray.c @@ -0,0 +1,211 @@ +// SPDX-License-Identifier: GPL-2.0+ +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define SMOX_CACHE_NAME "smox_test" +static struct kmem_cache *cachep; + +/* + * Declare XArrays globally so we can clean them up on module unload. + */ + +/* Used by test_smo_xarray()*/ +DEFINE_XARRAY(things); + +/* Thing to store pointers to in the XArray */ +struct smox_thing { + long id; +}; + +/* It's up to the caller to ensure id is unique */ +static struct smox_thing *alloc_thing(int id) +{ + struct smox_thing *thing; + + thing = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!thing) + return ERR_PTR(-ENOMEM); + + thing->id = id; + return thing; +} + +/** + * smox_object_ctor() - SMO object constructor function. + * @ptr: Pointer to memory where the object should be constructed. + */ +void smox_object_ctor(void *ptr) +{ + struct smox_thing *thing = ptr; + + thing->id = -1; +} + +/** + * smox_cache_migrate() - kmem_cache migrate function. + * @cp: kmem_cache pointer. + * @objs: Array of pointers to objects to migrate. + * @size: Number of objects in @objs. + * @node: NUMA node where the object should be allocated. + * @private: Pointer returned by kmem_cache_isolate_func(). + */ +void smox_cache_migrate(struct kmem_cache *cp, void **objs, int size, + int node, void *private) +{ + struct smox_thing **ptrs = (struct smox_thing **)objs; + struct smox_thing *old, *new; + struct smox_thing *thing; + unsigned long index; + void *entry; + int i; + + for (i = 0; i < size; i++) { + old = ptrs[i]; + + new = kmem_cache_alloc(cachep, GFP_KERNEL); + if (!new) { + pr_debug("kmem_cache_alloc failed\n"); + return; + } + + new->id = old->id; + + /* Update reference the brain dead way */ + xa_for_each(&things, index, thing) { + if (thing == old) { + entry = xa_store(&things, index, new, GFP_KERNEL); + if (entry != old) { + pr_err("failed to exchange new/old\n"); + return; + } + } + } + kmem_cache_free(cachep, old); + } +} + +/* + * test_smo_xarray() - Run some tests using an XArray. + */ +static int test_smo_xarray(void) +{ + const int keep = 6; /* Free 5 out of 6 items */ + const int nr_items = 10000; + struct smox_thing *thing; + unsigned long index; + void *entry; + int expected; + int i; + + /* + * Populate XArray, this adds to the radix_tree_node cache as + * well as the smox_test cache. + */ + for (i = 0; i < nr_items; i++) { + thing = alloc_thing(i); + entry = xa_store(&things, i, thing, GFP_KERNEL); + if (xa_is_err(entry)) { + pr_err("smox: failed to allocate entry: %d\n", i); + return -ENOMEM; + } + } + + /* Now free items, putting holes in both caches. */ + for (i = 0; i < nr_items; i++) { + if (i % keep == 0) + continue; + + thing = xa_erase(&things, i); + if (xa_is_err(thing)) + pr_err("smox: error erasing entry: %d\n", i); + kmem_cache_free(cachep, thing); + } + + expected = 0; + xa_for_each(&things, index, thing) { + if (thing->id != expected || index != expected) { + pr_err("smox: error; got %ld want %d at %ld\n", + thing->id, expected, index); + return -1; + } + expected += keep; + } + + /* + * Leave caches sparsely allocated. Shrink caches manually with: + * + * slabinfo radix_tree_node --shrink + * slabinfo smox_test --shrink + */ + + return 0; +} + +static int __init smox_cache_init(void) +{ + cachep = kmem_cache_create(SMOX_CACHE_NAME, + sizeof(struct smox_thing), + 0, 0, smox_object_ctor); + if (!cachep) + return -1; + + return 0; +} + +static void __exit smox_cache_cleanup(void) +{ + struct smox_thing *thing; + unsigned long i; + + xa_for_each(&things, i, thing) { + kmem_cache_free(cachep, thing); + } + xa_destroy(&things); + kmem_cache_destroy(cachep); +} + +static int __init smox_init(void) +{ + int ret; + + ret = smox_cache_init(); + if (ret) { + pr_err("smo_xarray: failed to create cache\n"); + return ret; + } + pr_info("smo_xarray: created kmem_cache: %s\n", SMOX_CACHE_NAME); + + kmem_cache_setup_mobility(cachep, NULL, smox_cache_migrate); + pr_info("smo_xarray: kmem_cache %s defrag enabled\n", SMOX_CACHE_NAME); + + /* + * Running this test consumes memory unless you shrink the + * radix_tree_node cache manually with `slabinfo`. + */ + ret = test_smo_xarray(); + if (ret) + pr_warn("test_smo_xarray failed: %d\n", ret); + + pr_info("smo_xarray: module loaded successfully\n"); + return 0; +} +module_init(smox_init); + +static void __exit smox_exit(void) +{ + smox_cache_cleanup(); + + pr_info("smo_xarray: module removed\n"); +} +module_exit(smox_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Tobin C. Harding"); +MODULE_DESCRIPTION("SMO XArray test module."); From patchwork Tue Apr 30 03:07:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922769 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9882B1515 for ; Tue, 30 Apr 2019 03:10:15 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 81F1A28646 for ; Tue, 30 Apr 2019 03:10:15 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7229328681; Tue, 30 Apr 2019 03:10:15 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7F15E28646 for ; Tue, 30 Apr 2019 03:10:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 752D16B027E; Mon, 29 Apr 2019 23:10:13 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 72AE66B027F; Mon, 29 Apr 2019 23:10:13 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F38E6B0280; Mon, 29 Apr 2019 23:10:13 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 3F89B6B027E for ; Mon, 29 Apr 2019 23:10:13 -0400 (EDT) Received: by mail-qt1-f199.google.com with SMTP id w53so12101861qtj.22 for ; Mon, 29 Apr 2019 20:10:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=YTpDBiPIz+r0uG1auSa8ph45UtBEpiDJlB6LNiHXZmQ=; b=Y3a4M7YT+WtlhKvFEhdja5pfm6O3L3J6iGPpuYXte1n1JzfFa8CSVLAIWn3/bbh5vD jUt0xaQ1AfSJC3RnwM6hQt6vHwzV5/cYFciRiqbSdQbu3W8RRqhRCFj+MmGSrrc06oya X/uHSW77GvpZYhubkTtYBDpqE6eLgXe1BwFDuC68V0WrYaZPWnDN+rv+gvu1KPKwbqBl /qhZ+wYiIFvlwHFKRUm3KLATKjzhVZJ39wnsj7ZgyaADuQsdcwOMbXTqzQxkzWN2xYu0 4QLSQkyP4zmNhtlpC4wEcBDbJyyC7jmDtEQT9xd0QGZRjihGaKF9ZT1+4cCUpc+oxhRc JMWQ== X-Gm-Message-State: APjAAAVavY9HpAs9abxqKzPRQP80cNM3fBGm40x8a1tl6Zow5GRtc3Kf cy6Y7o1DnnJ0ZHcPNRn+HpBXOLBtM4zcs/u9c8/LHo5vgn96W6fRwx1brjjkTjXzLD/zkaeojlh 1bcavvfcm1CjokI1yjr1AxcCAqymtKXpkyCUFX6G+CaRCQgEpo1w7hwq7O8Dcemw= X-Received: by 2002:a37:a1c4:: with SMTP id k187mr38220610qke.341.1556593812982; Mon, 29 Apr 2019 20:10:12 -0700 (PDT) X-Google-Smtp-Source: APXvYqxPVDy+WAc9Qg8dyZdSdxF806KmSnPXnitHOi8nEHsrL0IysFJwslWNItQQjzihs3tl+pIN X-Received: by 2002:a37:a1c4:: with SMTP id k187mr38220556qke.341.1556593811514; Mon, 29 Apr 2019 20:10:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593811; cv=none; d=google.com; s=arc-20160816; b=y9qRoV5j28Aqi9ws/4/0+a3FgB0NSQTXapxcst1SQHyKmWYM8p0m2Cd1xZU2fHeDii KRFvBNUx2qnSXfdy7GF12O0Cwtd5dH2eB2HrlbBYpsN+/obU6AmcBP+Zbnn79UDTGKsp L1ObrzH6+AvMSV7NQe0qjj4H9Qx0W+YdsICtE9c43pHNDRL/QP8vQDRZiK3TcIjVRmdQ AlYFaDF/CeSWl0Mdo1gHNHNVq9fmJF1z9WIZnoYFyMqPv7fWCAVQnbko3Wko/RyheC5O R8o2fNwgsUsDnGTFCizYtc7gqzm3ZvWomnLBsS4pmQ4NPQIhjECHkN5jHyypMmfaEAjB 6mBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=YTpDBiPIz+r0uG1auSa8ph45UtBEpiDJlB6LNiHXZmQ=; b=XKVqWskjkPgX4eiToOKr27+8aMJVUrJcfL4gvei0nNa+HJkhOyQgpySp9EzeTf/Ehf MZPagO1/xYEyfZV5eHjYNbp25Zdx2gq8hM+fTCsS/gCkkdd8jIAWySBJD1ZygoUajeSp BJhDiIRWSYyQ5SKYgiZwgBAmJsVN/qlpxTr0V3ul3nXXOGIa4HbZQJtcGmjRC50b8p+U EwSV12E2yeg6EQRXUS6zHqb2bMGRl8pWrPyfEm6itPhhJU7Bg4w+Vx2rVJ0pNjRZFKK4 3UjY5uJ31HcKWvtKom4wfq04Ho90ZWHyJfM/p8unRVvdZqEd0PBGQrXUMKDvEArOnF0e 3xMQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=Gp5AtuwP; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id i9si193153qkg.149.2019.04.29.20.10.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:10:11 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=Gp5AtuwP; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 34C78A460; Mon, 29 Apr 2019 23:10:11 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:10:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=YTpDBiPIz+r0uG1auSa8ph45UtBEpiDJlB6LNiHXZmQ=; b=Gp5AtuwP zsytgvXMsqqx/PEtOwlH2cf2yZ3KT6uRfusGcuJ2FfuUcZKoYIEYwFCV4q1J0Ilr kYc/XwcdafDBxRD/DxKrSDtaO144R64GZgzlTWI4u6zCBTJqVGeKvyLzdjxBa6IT sHzUCFtM7mHHGVcFgmawPMJFu8VqeFrbVxvG+AalTTxjBn2haouwmmqTOoCTtyBv KqBuTy6y3zpytrZdsA4+wR9FKQoDPi1xHG6rVZIW1Uz2EFPushPjUPHjS2Rj1PrG MO1r5tX/r5UA27ehob5lmdb5fTOBJJaCGmARSXhGwwXXRcBRoYDMfI0/DKKvq2bw lADQ4WHAzmGabQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeikecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedutd X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 072E5103C8; Mon, 29 Apr 2019 23:10:02 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 11/15] slub: Enable moving objects to/from specific nodes Date: Tue, 30 Apr 2019 13:07:42 +1000 Message-Id: <20190430030746.26102-12-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We have just implemented Slab Movable Objects (object migration). Currently object migration is used to defrag a cache. On NUMA systems it would be nice to be able to control the source and destination nodes when moving objects. Add CONFIG_SMO_NODE to guard this feature. CONFIG_SMO_NODE depends on CONFIG_SLUB_DEBUG because we use the full list. Leave it like this for the RFC because the patch will be less cluttered to review, separate full list out of CONFIG_DEBUG before doing a PATCH version. Implement moving all objects (including those in full slabs) to a specific node. Expose this functionality to userspace via a sysfs entry. Add sysfs entry: /sysfs/kernel/slab//move With this users get access to the following functionality: - Move all objects to specified node. echo "N1" > move - Move all objects from specified node to other specified node (from N1 -> to N2): echo "N1 N2" > move This also enables shrinking slabs on a specific node: echo "N1 N1" > move Signed-off-by: Tobin C. Harding --- mm/Kconfig | 7 ++ mm/slub.c | 249 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 256 insertions(+) diff --git a/mm/Kconfig b/mm/Kconfig index 25c71eb8a7db..47040d939f3b 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -258,6 +258,13 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION config ARCH_ENABLE_THP_MIGRATION bool +config SMO_NODE + bool "Enable per node control of Slab Movable Objects" + depends on SLUB && SYSFS + select SLUB_DEBUG + help + On NUMA systems enable moving objects to and from a specified node. + config PHYS_ADDR_T_64BIT def_bool 64BIT diff --git a/mm/slub.c b/mm/slub.c index e601c804ed79..e4f3dde443f5 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4345,6 +4345,106 @@ static void move_slab_page(struct page *page, void *scratch, int node) s->migrate(s, vector, count, node, private); } +#ifdef CONFIG_SMO_NODE +/* + * kmem_cache_move() - Attempt to move all slab objects. + * @s: The cache we are working on. + * @node: The node to move objects away from. + * @target_node: The node to move objects on to. + * + * Attempts to move all objects (partial slabs and full slabs) to target + * node. + * + * Context: Takes the list_lock. + * Return: The number of slabs remaining on node. + */ +static unsigned long kmem_cache_move(struct kmem_cache *s, + int node, int target_node) +{ + struct kmem_cache_node *n = get_node(s, node); + LIST_HEAD(move_list); + struct page *page, *page2; + unsigned long flags; + void **scratch; + + if (!s->migrate) { + pr_warn("%s SMO not enabled, cannot move objects\n", s->name); + goto out; + } + + scratch = alloc_scratch(s); + if (!scratch) + goto out; + + spin_lock_irqsave(&n->list_lock, flags); + + list_for_each_entry_safe(page, page2, &n->partial, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + if (page->inuse) { + list_move(&page->lru, &move_list); + /* Stop page being considered for allocations */ + n->nr_partial--; + page->frozen = 1; + + slab_unlock(page); + } else { /* Empty slab page */ + list_del(&page->lru); + n->nr_partial--; + slab_unlock(page); + discard_slab(s, page); + } + } + list_for_each_entry_safe(page, page2, &n->full, lru) { + if (!slab_trylock(page)) + continue; + + list_move(&page->lru, &move_list); + page->frozen = 1; + slab_unlock(page); + } + + spin_unlock_irqrestore(&n->list_lock, flags); + + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + + /* Bail here to save taking the list_lock */ + if (list_empty(&move_list)) + goto out; + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + if (page->inuse == page->objects) { + list_add(&page->lru, &n->full); + slab_unlock(page); + } else { + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); +out: + return atomic_long_read(&n->nr_slabs); +} +#endif /* CONFIG_SMO_NODE */ + /* * kmem_cache_defrag() - Defragment node. * @s: cache we are working on. @@ -4459,6 +4559,32 @@ static unsigned long kmem_cache_defrag(struct kmem_cache *s, return n->nr_partial; } +#ifdef CONFIG_SMO_NODE +/* + * kmem_cache_move_to_node() - Move all slab objects to node. + * @s: The cache we are working on. + * @node: The target node to move objects to. + * + * Attempt to move all slab objects from all nodes to @node. + * + * Return: The total number of slabs left on emptied nodes. + */ +static unsigned long kmem_cache_move_to_node(struct kmem_cache *s, int node) +{ + unsigned long left = 0; + int nid; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == node) + continue; + + left += kmem_cache_move(s, nid, node); + } + + return left; +} +#endif + /** * kmem_defrag_slabs() - Defrag slab caches. * @node: The node to defrag or -1 for all nodes. @@ -5603,6 +5729,126 @@ static ssize_t shrink_store(struct kmem_cache *s, } SLAB_ATTR(shrink); +#ifdef CONFIG_SMO_NODE +static ssize_t move_show(struct kmem_cache *s, char *buf) +{ + return 0; +} + +/* + * parse_move_store_input() - Parse buf getting integer arguments. + * @buf: Buffer to parse. + * @length: Length of @buf. + * @arg0: Return parameter, first argument. + * @arg1: Return parameter, second argument. + * + * Parses the input from user write to sysfs file 'move'. Input string + * should contain either one or two node specifiers of form Nx where x + * is an integer specifying the NUMA node ID. 'N' or 'n' may be used. + * n/N may be omitted. + * + * e.g. + * echo 'N1' > /sysfs/kernel/slab/cache/move + * or + * echo 'N0 N2' > /sysfs/kernel/slab/cache/move + * + * Regex matching accepted forms: '[nN]?[0-9]( [nN]?[0-9])?' + * + * FIXME: This is really fragile. Input must be exactly correct, + * spurious whitespace causes parse errors. + * + * Return: 0 if an argument was successfully converted, or an error code. + */ +static ssize_t parse_move_store_input(const char *buf, size_t length, + long *arg0, long *arg1) +{ + char *s, *save, *ptr; + int ret = 0; + + if (!buf) + return -EINVAL; + + s = kstrdup(buf, GFP_KERNEL); + if (!s) + return -ENOMEM; + save = s; + + if (s[length - 1] == '\n') { + s[length - 1] = '\0'; + length--; + } + + ptr = strsep(&s, " "); + if (!ptr || strcmp(ptr, "") == 0) { + ret = 0; + goto out; + } + + if (*ptr == 'N' || *ptr == 'n') + ptr++; + ret = kstrtol(ptr, 10, arg0); + if (ret < 0) + goto out; + + if (s) { + if (*s == 'N' || *s == 'n') + s++; + ret = kstrtol(s, 10, arg1); + if (ret < 0) + goto out; + } + + ret = 0; +out: + kfree(save); + return ret; +} + +static bool is_valid_node(int node) +{ + int nid; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == node) + return true; + } + return false; +} + +/* + * move_store() - Move objects between nodes. + * @s: The cache we are working on. + * @buf: String received. + * @length: Length of @buf. + * + * Writes to /sys/kernel/slab//move are interpreted as follows: + * + * echo "N1" > move : Move all objects (from all nodes) to node 1. + * echo "N0 N1" > move : Move all objects from node 0 to node 1. + * + * 'N' may be omitted: + */ +static ssize_t move_store(struct kmem_cache *s, const char *buf, size_t length) +{ + long arg0 = -1; + long arg1 = -1; + int ret; + + ret = parse_move_store_input(buf, length, &arg0, &arg1); + if (ret < 0) + return -EINVAL; + + if (is_valid_node(arg0) && is_valid_node(arg1)) + (void)kmem_cache_move(s, arg0, arg1); + else if (is_valid_node(arg0)) + (void)kmem_cache_move_to_node(s, arg0); + + /* FIXME: What should we be returning here? */ + return length; +} +SLAB_ATTR(move); +#endif /* CONFIG_SMO_NODE */ + #ifdef CONFIG_NUMA static ssize_t remote_node_defrag_ratio_show(struct kmem_cache *s, char *buf) { @@ -5727,6 +5973,9 @@ static struct attribute *slab_attrs[] = { &reclaim_account_attr.attr, &destroy_by_rcu_attr.attr, &shrink_attr.attr, +#ifdef CONFIG_SMO_NODE + &move_attr.attr, +#endif &slabs_cpu_partial_attr.attr, #ifdef CONFIG_SLUB_DEBUG &total_objects_attr.attr, From patchwork Tue Apr 30 03:07:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922773 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E5EB1390 for ; Tue, 30 Apr 2019 03:10:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1D13F289BF for ; Tue, 30 Apr 2019 03:10:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 10E3028A4E; Tue, 30 Apr 2019 03:10:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58834289BF for ; Tue, 30 Apr 2019 03:10:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E0A56B0282; Mon, 29 Apr 2019 23:10:22 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 492A56B0283; Mon, 29 Apr 2019 23:10:22 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3320B6B0284; Mon, 29 Apr 2019 23:10:22 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id 1209D6B0282 for ; Mon, 29 Apr 2019 23:10:22 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id k6so10741705qkf.13 for ; Mon, 29 Apr 2019 20:10:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=aa6iExGIwghZDn1ZohvQXbuCEdzySOldf4bort+1eX4=; b=c4vjmygDxNgPoabrEig7WAHWWk1cpxB2NNWphxYU56H7+J7tZJPIfN620Q6kfbpULv RcaCiqaG3Tj19UgaupsegXaUvKEzkfykM04ofDFUO+fg/RlawZIMsVNyQZv7bJdH0so0 MaTntrcqOXyenBJ26xq/RqaMDlOEC/vZ6ugECT5YIvYsqCj32CLvFCsK2I8/dasc2SPT bz/d+eoeguXJZUvKUTISzXMlSI4beyd4d0OE90qzwxuEoQSy09c1sQS1xyUQftlq/5T/ VCZnitMhpk4nWme6qHi6eZPA3+QiEbZEWVp37qBi2Mo0Vzz4USi86S6BoNJ6WU/APEDS tnsA== X-Gm-Message-State: APjAAAVbVSshzgjwZgy1tdkUZGZCMDjNijHMXPnfpPKdZRs1tK+VUq74 jvmyG/Tlo2NrPtCGN7Zzmdrf3vTDmesFQOL4ZyioYCjrKynA4IxsNgAmHAV+2BjA+iFu9jFUtym I2G9a5W3hqzMiocMaj2rHS8Uj4wtVD8Ex4+53jGGbgoTeszT19lEtPUHyLKNRNOg= X-Received: by 2002:a0c:d603:: with SMTP id c3mr41467243qvj.144.1556593821831; Mon, 29 Apr 2019 20:10:21 -0700 (PDT) X-Google-Smtp-Source: APXvYqw6VHNMbuiA2hrvASaLpgbpcBiv81vjHaxAVRg9DHKyhndhRuUGZDRwugsB+DRBbeMFYfw3 X-Received: by 2002:a0c:d603:: with SMTP id c3mr41467197qvj.144.1556593820542; Mon, 29 Apr 2019 20:10:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593820; cv=none; d=google.com; s=arc-20160816; b=I4MvaAfzyMdheLtCSr/AIoTyVvool4PPBrhEWhUiBaRms+v8ZBUhv1/oZkNSYlLTBY bi4nQmMyMJglkMwVm3Jwq/N8n5Y7TQVLxArQOJQ6LQbD9YJYyLNR5JtB8DDD+4e4NQ65 WbbloFP1hzG6HbHfiBaJHh2XksTRonOXxA4hWAE3YtfTeZ1wtWBZRfVglCnS7JFqGdr2 aNXQ0M2Me9w/Ge2y9Rfp8PvLux2OBWxeJrAbt4JHXiXU8Oz9tpkfl0VxK37SUj/TKuQC QqxJMZy5NWODUityrpPH1KHWsAejJd3mLL6Fx7qZvgkT31e1t8o4cGjjwbaWEvnYKRE6 dIdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=aa6iExGIwghZDn1ZohvQXbuCEdzySOldf4bort+1eX4=; b=CMiZu4kwKMdgFS3Qb7eOeoe7JZCqnN/I21rS9JTeU48k+Zw51JqKJWRDu0REBeXTAQ 5okhOfy0tlrtvkHKwuFmtC92MMph1S1CwOcGAXjcPdqG9fb6XHsma7XJf7nF/HwoILYc SOAkbOxGN4WhEhuHgXJ9lODTP9zpW+zjvhMKpXFXhGR18H/5V5fOmGmuuvuXIUzVFLj2 GSgau2IOK8JjBoQSkse4BIPyXycrcex1jiKIftB3AZfczaA2j/fnYAoZ0TqcJ5Aoc8+s tIei5QPKk5GDpVq8y+SnquPkv0GqN85xF7eGfTlaFYQNMAhDjTbHghuI4+3OnwxI32oC AbuA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=eq8zJ4mn; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id m129si812870qkd.2.2019.04.29.20.10.20 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:10:20 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=eq8zJ4mn; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 48A29B24E; Mon, 29 Apr 2019 23:10:20 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:10:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=aa6iExGIwghZDn1ZohvQXbuCEdzySOldf4bort+1eX4=; b=eq8zJ4mn Lf3zb/snoc2FYDatoMhTLJ9Q4aZfQKL3GjXL3f59hF9pGLS0ly2Iw8VD6yKgIaMn fm9GLilO8ToenemuTBE3wLNe7QcfqeKyr7kAXc3UfNm01VqP6idcDVfOxgQFfRg3 3MbjdgyWaveCL1W/BNxVwnpVSymreHU8+WLRqP1T8Y4flfpDv5NEZQ5J0vm4jg0X XPqjWCChAE+FucXI639pKS4FT9u/kePz7Y+rIakvO4vc3jmU1TPvGTDv6Y7Vy5hI +G6OjX+4/5eiHB1QzWvBcAFM12mGZi/Qpx6EmRGlL+eIJTlz54oP1+iGXdYCqSWf vsNbyM7NeoNnfw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeilecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 855FF103C9; Mon, 29 Apr 2019 23:10:11 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 12/15] slub: Enable balancing slabs across nodes Date: Tue, 30 Apr 2019 13:07:43 +1000 Message-Id: <20190430030746.26102-13-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP We have just implemented Slab Movable Objects (SMO). On NUMA systems slabs can become unbalanced i.e. many slabs on one node while other nodes have few slabs. Using SMO we can balance the slabs across all the nodes. The algorithm used is as follows: 1. Move all objects to node 0 (this has the effect of defragmenting the cache). 2. Calculate the desired number of slabs for each node (this is done using the approximation nr_slabs / nr_nodes). 3. Loop over the nodes moving the desired number of slabs from node 0 to the node. Feature is conditionally built in with CONFIG_SMO_NODE, this is because we need the full list (we enable SLUB_DEBUG to get this). Future version may separate final list out of SLUB_DEBUG. Expose this functionality to userspace via a sysfs entry. Add sysfs entry: /sysfs/kernel/slab//balance Write of '1' to this file triggers balance, no other value accepted. This feature relies on SMO being enable for the cache, this is done with a call to, after the isolate/migrate functions have been defined. kmem_cache_setup_mobility(s, isolate, migrate) Signed-off-by: Tobin C. Harding --- mm/slub.c | 120 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) diff --git a/mm/slub.c b/mm/slub.c index e4f3dde443f5..a5c48c41d72b 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4583,6 +4583,109 @@ static unsigned long kmem_cache_move_to_node(struct kmem_cache *s, int node) return left; } + +/* + * kmem_cache_move_slabs() - Attempt to move @num slabs to target_node, + * @s: The cache we are working on. + * @node: The node to move objects from. + * @target_node: The node to move objects to. + * @num: The number of slabs to move. + * + * Attempts to move @num slabs from @node to @target_node. This is done + * by migrating objects from slabs on the full_list. + * + * Return: The number of slabs moved or error code. + */ +static long kmem_cache_move_slabs(struct kmem_cache *s, + int node, int target_node, long num) +{ + struct kmem_cache_node *n = get_node(s, node); + LIST_HEAD(move_list); + struct page *page, *page2; + unsigned long flags; + void **scratch; + long done = 0; + + if (node == target_node) + return -EINVAL; + + scratch = alloc_scratch(s); + if (!scratch) + return -ENOMEM; + + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &n->full, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + list_move(&page->lru, &move_list); + page->frozen = 1; + slab_unlock(page); + + if (++done >= num) + break; + } + spin_unlock_irqrestore(&n->list_lock, flags); + + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + move_slab_page(page, scratch, target_node); + } + kfree(scratch); + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + /* + * This is best effort only, if slab still has + * objects just put it back on the partial list. + */ + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); + + return done; +} + +/* + * kmem_cache_balance_nodes() - Balance slabs across nodes. + * @s: The cache we are working on. + */ +static void kmem_cache_balance_nodes(struct kmem_cache *s) +{ + struct kmem_cache_node *n = get_node(s, 0); + unsigned long desired_nr_slabs_per_node; + unsigned long nr_slabs; + int nr_nodes = 0; + int nid; + + (void)kmem_cache_move_to_node(s, 0); + + for_each_node_state(nid, N_NORMAL_MEMORY) + nr_nodes++; + + nr_slabs = atomic_long_read(&n->nr_slabs); + desired_nr_slabs_per_node = nr_slabs / nr_nodes; + + for_each_node_state(nid, N_NORMAL_MEMORY) { + if (nid == 0) + continue; + + kmem_cache_move_slabs(s, 0, nid, desired_nr_slabs_per_node); + } +} #endif /** @@ -5847,6 +5950,22 @@ static ssize_t move_store(struct kmem_cache *s, const char *buf, size_t length) return length; } SLAB_ATTR(move); + +static ssize_t balance_show(struct kmem_cache *s, char *buf) +{ + return 0; +} + +static ssize_t balance_store(struct kmem_cache *s, + const char *buf, size_t length) +{ + if (buf[0] == '1') + kmem_cache_balance_nodes(s); + else + return -EINVAL; + return length; +} +SLAB_ATTR(balance); #endif /* CONFIG_SMO_NODE */ #ifdef CONFIG_NUMA @@ -5975,6 +6094,7 @@ static struct attribute *slab_attrs[] = { &shrink_attr.attr, #ifdef CONFIG_SMO_NODE &move_attr.attr, + &balance_attr.attr, #endif &slabs_cpu_partial_attr.attr, #ifdef CONFIG_SLUB_DEBUG From patchwork Tue Apr 30 03:07:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922779 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F36971390 for ; Tue, 30 Apr 2019 03:10:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E40A628994 for ; Tue, 30 Apr 2019 03:10:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D7F68289C9; Tue, 30 Apr 2019 03:10:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 680E628994 for ; Tue, 30 Apr 2019 03:10:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6417E6B0285; Mon, 29 Apr 2019 23:10:30 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5F22C6B0286; Mon, 29 Apr 2019 23:10:30 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BAC36B0287; Mon, 29 Apr 2019 23:10:30 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 2E5226B0285 for ; Mon, 29 Apr 2019 23:10:30 -0400 (EDT) Received: by mail-qt1-f199.google.com with SMTP id q57so12123837qtf.11 for ; Mon, 29 Apr 2019 20:10:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=xGuBBSB886petOw3w98G/aZf3m1f0zKs3QEeiyyCCSU=; b=C690kEfBs6r7QiTrmxjgatP72DRiKjdqYBzTysWhbJf1REIHg8dEeI2hrBTt8Y5dT8 3S2/UZrY8pLlGk71d7iVB869f9yXtU8gCVHQ/IjxnVmAfY8bKBXnlRrHQJ0lBdx9mjOc Mi6KTRy4oEe2LMPqIfTmiaJz+66KYlXIRJcLXtmABrtrLfyzTIu0I6jnTPn2o/Mj1JHp gTIcifmuJ23CwenqnEk6bz9R+IoApr2Aj5jpZcUXww8UUKFXjf+ASEgRnGyRx0+UMJqo fTJK7YMYOlCO9pZ8uPBYB/JDhbwQp9aRSGBMjK1/tyeC2PCSWUQjEmMSCHVY99Z9Ty/v KdNQ== X-Gm-Message-State: APjAAAWDqx9uhUSFvYtsUgKtdWdYNB6UeCHbIW9RyJ76IcRIDL6RJRHm RoG9NTnvVdB2Vxjm9oKYRtNcNKt0+H9WFnW9mKXe4Mo7SHOiQVKD3yT4qVZ0i4OH5srDxTn9BAm AJW6aLH4zGXq7FOSNSFaLUzMJiAptAYQV74KKF+6TYcmeE3lHb2cng9NB5eNG2a0= X-Received: by 2002:a37:ef13:: with SMTP id j19mr29155874qkk.264.1556593829961; Mon, 29 Apr 2019 20:10:29 -0700 (PDT) X-Google-Smtp-Source: APXvYqznLK1M82EmxDlm1Xk87WakWTyurZBvvqfszyRGhWAzjDOUIs1FWTv9pYiSTgaD6dlBPRf8 X-Received: by 2002:a37:ef13:: with SMTP id j19mr29155827qkk.264.1556593828797; Mon, 29 Apr 2019 20:10:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593828; cv=none; d=google.com; s=arc-20160816; b=1D5rQq5DlEVx5OTT3ldYorb1GTYkClOYXBs5dve3Kw7rd9h2dxhVyahFwo9ABUE+sM dk4E8D8RlDRW29/qJrBjFEzGvbZZj0NgQmqthDitYsoiJlabLsf3jy+qTr/seQnxD1KO /nIl9QyaGVtmUro/hTuig+dXNFpg9merkwguNEwFuDgbl+meUCoG5sb3p0wPdJ9svrtx ioTtfpmL5ff2dKq9RxaywQPvNkTG+sIJP4CScMpbB6YWy6bKBge06bWytcuI6MyiU3vc VSzPgsP8fQ/cNBkad4BwNvPm7cvAVS7HXp+zFrdzFx0yNuD4hbxyd1taolJdYpjgWhZh mH7Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=xGuBBSB886petOw3w98G/aZf3m1f0zKs3QEeiyyCCSU=; b=eBBe65xUU0oaVKY7h5qs5uzSFqvq54tl/HONjGMogxN4fUFfaWBf1bX6JvdlNkfmr5 jvVrFWHGa7xVzqRHQVeUXjeCtaOi+G64KkY153j0eSFYsy13g7DmcE1BNLdY0U3E4ZCJ mBiwdtO+sogVCnYjR6w3gox39b0Yq9lN34hZjYmn3uzxJ8G0RIPFpsYptOi9YRumx22w EvIkBeQpZXuvhFyWeRNsMXao34hY4NWfftv54s4iG7/fbv91EIgQxYBxN/+LA1143NEN 4vnSqtdPDsczis2NTfI+nazWsw0RiW67fuY5+83s9obkgE1P2/d0fZKI9Xh7tQBhlBkl 470w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ApONlV9R; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id e18si7436331qkg.90.2019.04.29.20.10.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:10:28 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=ApONlV9R; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id 8548FB542; Mon, 29 Apr 2019 23:10:28 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:10:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=xGuBBSB886petOw3w98G/aZf3m1f0zKs3QEeiyyCCSU=; b=ApONlV9R CYbP8MsPKwD89JCHecL+ibpiKvLaTyC2MjKvGMDLUPLGQTpzHcs27o4mzHiysfWC n7hDZB4eGPgwtwxsbVQ5W7t4HZzVZ/B7n0tpi0HpJn+VjqewjCoqt5eZphGvfhZb 5uthRkIF5WbSgYA4ftlq5s/tFppzFc5BeU2WgvdpDJn+L5M/WJFBC9Du7qoukB1Y w2s23iqJ97QC0GYwqT3wgHCGH68Mg7bZVYXGeiZ2o7I5bYlP27wQ78XnG2dianoc m5TBnA6jyWPOz4isWQA7Qe0TMaby+2K9WTIAqVxx6JVxPNsbAoNJ/wJpcOvQLB8U bbo89/JC653vwg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeilecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedu X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 916CF103C8; Mon, 29 Apr 2019 23:10:20 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 13/15] dcache: Provide a dentry constructor Date: Tue, 30 Apr 2019 13:07:44 +1000 Message-Id: <20190430030746.26102-14-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In order to support object migration on the dentry cache we need to have a determined object state at all times. Without a constructor the object would have a random state after allocation. Provide a dentry constructor. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index aac41adf4743..3d6cc06eca56 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -1603,6 +1603,16 @@ void d_invalidate(struct dentry *dentry) } EXPORT_SYMBOL(d_invalidate); +static void dcache_ctor(void *p) +{ + struct dentry *dentry = p; + + /* Mimic lockref_mark_dead() */ + dentry->d_lockref.count = -128; + + spin_lock_init(&dentry->d_lock); +} + /** * __d_alloc - allocate a dcache entry * @sb: filesystem it will belong to @@ -1658,7 +1668,6 @@ struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) dentry->d_lockref.count = 1; dentry->d_flags = 0; - spin_lock_init(&dentry->d_lock); seqcount_init(&dentry->d_seq); dentry->d_inode = NULL; dentry->d_parent = dentry; @@ -3091,14 +3100,17 @@ static void __init dcache_init_early(void) static void __init dcache_init(void) { - /* - * A constructor could be added for stable state like the lists, - * but it is probably not worth it because of the cache nature - * of the dcache. - */ - dentry_cache = KMEM_CACHE_USERCOPY(dentry, - SLAB_RECLAIM_ACCOUNT|SLAB_PANIC|SLAB_MEM_SPREAD|SLAB_ACCOUNT, - d_iname); + slab_flags_t flags = + SLAB_RECLAIM_ACCOUNT | SLAB_PANIC | SLAB_MEM_SPREAD | SLAB_ACCOUNT; + + dentry_cache = + kmem_cache_create_usercopy("dentry", + sizeof(struct dentry), + __alignof__(struct dentry), + flags, + offsetof(struct dentry, d_iname), + sizeof_field(struct dentry, d_iname), + dcache_ctor); /* Hash may have been set up in dcache_init_early */ if (!hashdist) From patchwork Tue Apr 30 03:07:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922781 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 623831515 for ; Tue, 30 Apr 2019 03:10:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 50D5728994 for ; Tue, 30 Apr 2019 03:10:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 44571289C9; Tue, 30 Apr 2019 03:10:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AA7ED28994 for ; Tue, 30 Apr 2019 03:10:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BE8096B0288; Mon, 29 Apr 2019 23:10:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BBE276B028A; Mon, 29 Apr 2019 23:10:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A88416B0288; Mon, 29 Apr 2019 23:10:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id 89BCE6B0288 for ; Mon, 29 Apr 2019 23:10:38 -0400 (EDT) Received: by mail-qk1-f198.google.com with SMTP id a12so10767484qkb.3 for ; Mon, 29 Apr 2019 20:10:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=AtTOS3tLSFxiBVY6HKqiEZQDmJwYtvFPI1Eb7X0+v6o=; b=o7VIEU2QXX5t3MH2ksZYHV0fYVL8qWN8AQy2+MtFTU+pcEq6Hfp9v2W3x0mq5jBJ8Q qpOSvr9JeCGVOqyFFfUyUk7g91PvesWpFTVL0OWwwNGIvcxt8XTY2Q5YQsJNiTLFkpp9 XOJyNPXcHpTHpR5gPh3+z9GE6eEebhijgxywRpYyAlhN0NzVrvU0T23mZ+sGGK3z7sX/ adnyMzQg+t898ijyCx6Hygig8p1/8wE3mkbxN0Pu0sctkEtWBQYnm3u13AcQ704fvPFX e2XpKYFE4lc2vLJPu1vfF6DtBUYdpp5M7BFAsChv6iEMIYDJiXZquGmbAZ+IB1S4jbgB gFIA== X-Gm-Message-State: APjAAAVTOFmzIeMiMdMsZYSkGdru+JSgWn8hhAwOeQq+L9iKSjbTwHEm Mn61tnO0f4o4WvFaQ26Db13XDIlSIeCAWZycfajtCJPelzcabz9r2CApdYuP2zL1cOUMTUDhl2U xCUZPgvXixSOoJLcxRu6Ppb033gSz6V9RAGttzdsahwTG5aXUMWrZNM7WRLXhB2k= X-Received: by 2002:a0c:87bb:: with SMTP id 56mr13722537qvj.219.1556593838305; Mon, 29 Apr 2019 20:10:38 -0700 (PDT) X-Google-Smtp-Source: APXvYqz3Av/a9DG8OEdW/2JETZt8rPiwwrsrcCnB6Bqh+E6Kxzc9zhWMBz+1NGz+qDdVEJe6U6Fz X-Received: by 2002:a0c:87bb:: with SMTP id 56mr13722498qvj.219.1556593837127; Mon, 29 Apr 2019 20:10:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593837; cv=none; d=google.com; s=arc-20160816; b=GZUQE1D7C9BhVV5QDogbxG1K9EXeJ8vUVvCVFKZcPHVspB6JbDdAC4r/L/cotJtvVt i8+INGOMGq0pI+nqyTu6dAjdgLljYYFFhj8AiIiI+Ln12wSLxy/DQBYA+ZuXuyRUBo7A 9s+QOc9SSXrVov7bwS4tPJ0EjoZBNUwA/yRZFvDtk1aKytxBxPUgriDB1qYGggyclp3/ 5WiL9ENTcZQekDtnYvzkjTPqTpKMeCPD2yoEgKC4Z2zF05G7drqXRQWtA48YC9sZM3nb bs0LVh+LzVPBZScq9cejeh3s9+618EYR/IPNFuALLx0FaAW/RPB+bJY24oC4EbXeqT9B bZqA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=AtTOS3tLSFxiBVY6HKqiEZQDmJwYtvFPI1Eb7X0+v6o=; b=bI5liHV7ECLq5obXitzF8lX3+DS5cAm1nkk3thpQ2bmGzhkzEeH7F6CUe9AXoxHjw0 Vnb7jXjGJeN7Vu0o1toxPzjdXPHo8HWgptKLLe2fSmy9CxTb/LfbOkl/xrA42rEykuid 8i5tL0+HDPY3cOpdV29iVyKK87Gpyfjuu95/Xh1hp5/HLLhvc/v+8zcn6Cul0y7tUljI FPm5zLgzJzykDMaG57Jpwp0NhfXCnG6tUX9/8OvNha5JtJQySx++WKP0Bzg5evGn1hFq edEiuvAq+7tThgbRrFZgOrFH7z8idle+yq45mH7ByjOolKbbdefQH3Nob78CqEm4+tlO fy9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=Mz0JYE8q; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id k8si6530946qvj.219.2019.04.29.20.10.37 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:10:37 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=Mz0JYE8q; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id D25E41489D; Mon, 29 Apr 2019 23:10:36 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:10:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=AtTOS3tLSFxiBVY6HKqiEZQDmJwYtvFPI1Eb7X0+v6o=; b=Mz0JYE8q 6mfgLxTsFaZvnrjTOzIU1K9OLF5xaipxeaIRg07vVAIGydkzG41j0c9cIjAaqfPu suuks90VIcxBWrW8xNvU/djcyBGQwdRU8LhIOmphBHp/qUEnucLkh9SU3+9ZrL3d v86+RFNKYWgLRshebiLPhu1k9CgZjaQD1CV3uvfH9jdUJwTpHIVG5GyAVb5fJUDT JmLxKIFRmxu8gZGpBn9e4PvBsthjqhArLsJfe0HayxvLV9XjrLsyETpCi35zzCbV 1SPFpp+quxv5q+CrN0Tuy0K7BDEPf39g70zWVvBkBJXP2T0/7SihlY93d07n60Fm ApjoRKAP/cCWfA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeilecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedv X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id 24FE9103C8; Mon, 29 Apr 2019 23:10:28 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 14/15] dcache: Implement partial shrink via Slab Movable Objects Date: Tue, 30 Apr 2019 13:07:45 +1000 Message-Id: <20190430030746.26102-15-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The dentry slab cache is susceptible to internal fragmentation. Now that we have Slab Movable Objects we can attempt to defragment the dcache. Dentry objects are inherently _not_ relocatable however under some conditions they can be free'd. This is the same as shrinking the dcache but instead of shrinking the whole cache we only attempt to free those objects that are located in partially full slab pages. There is no guarantee that this will reduce the memory usage of the system, it is a compromise between fragmented memory and total cache shrinkage with the hope that some memory pressure can be alleviated. This is implemented using the newly added Slab Movable Objects infrastructure. The dcache 'migration' function is intentionally _not_ called 'd_migrate' because we only free, we do not migrate. Call it 'd_partial_shrink' to make explicit that no reallocation is done. Implement isolate and 'migrate' functions for the dentry slab cache. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/fs/dcache.c b/fs/dcache.c index 3d6cc06eca56..3f9daba1cc78 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -30,6 +30,7 @@ #include #include #include +#include #include "internal.h" #include "mount.h" @@ -3067,6 +3068,79 @@ void d_tmpfile(struct dentry *dentry, struct inode *inode) } EXPORT_SYMBOL(d_tmpfile); +/* + * d_isolate() - Dentry isolation callback function. + * @s: The dentry cache. + * @v: Vector of pointers to the objects to isolate. + * @nr: Number of objects in @v. + * + * The slab allocator is holding off frees. We can safely examine + * the object without the danger of it vanishing from under us. + */ +static void *d_isolate(struct kmem_cache *s, void **v, int nr) +{ + struct list_head *dispose; + struct dentry *dentry; + int i; + + dispose = kmalloc(sizeof(*dispose), GFP_KERNEL); + if (!dispose) + return NULL; + + INIT_LIST_HEAD(dispose); + + for (i = 0; i < nr; i++) { + dentry = v[i]; + spin_lock(&dentry->d_lock); + + if (dentry->d_lockref.count > 0 || + dentry->d_flags & DCACHE_SHRINK_LIST) { + spin_unlock(&dentry->d_lock); + continue; + } + + if (dentry->d_flags & DCACHE_LRU_LIST) + d_lru_del(dentry); + + d_shrink_add(dentry, dispose); + spin_unlock(&dentry->d_lock); + } + + return dispose; +} + +/* + * d_partial_shrink() - Dentry migration callback function. + * @s: The dentry cache. + * @_unused: We do not access the vector. + * @__unused: No need for length of vector. + * @___unused: We do not do any allocation. + * @private: list_head pointer representing the shrink list. + * + * Dispose of the shrink list created during isolation function. + * + * Dentry objects can _not_ be relocated and shrinking the whole dcache + * can be expensive. This is an effort to free dentry objects that are + * stopping slab pages from being free'd without clearing the whole dcache. + * + * This callback is called from the SLUB allocator object migration + * infrastructure in attempt to free up slab pages by freeing dentry + * objects from partially full slabs. + */ +static void d_partial_shrink(struct kmem_cache *s, void **_unused, int __unused, + int ___unused, void *private) +{ + struct list_head *dispose = private; + + if (!private) /* kmalloc error during isolate. */ + return; + + if (!list_empty(dispose)) + shrink_dentry_list(dispose); + + kfree(private); +} + static __initdata unsigned long dhash_entries; static int __init set_dhash_entries(char *str) { @@ -3112,6 +3186,8 @@ static void __init dcache_init(void) sizeof_field(struct dentry, d_iname), dcache_ctor); + kmem_cache_setup_mobility(dentry_cache, d_isolate, d_partial_shrink); + /* Hash may have been set up in dcache_init_early */ if (!hashdist) return; From patchwork Tue Apr 30 03:07:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10922785 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C6A11390 for ; Tue, 30 Apr 2019 03:10:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D59F28994 for ; Tue, 30 Apr 2019 03:10:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6126A289C9; Tue, 30 Apr 2019 03:10:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF7AD28994 for ; Tue, 30 Apr 2019 03:10:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E308E6B028B; Mon, 29 Apr 2019 23:10:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D91D66B028C; Mon, 29 Apr 2019 23:10:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C80FE6B028D; Mon, 29 Apr 2019 23:10:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by kanga.kvack.org (Postfix) with ESMTP id 9E57B6B028B for ; Mon, 29 Apr 2019 23:10:46 -0400 (EDT) Received: by mail-qk1-f197.google.com with SMTP id c2so10734142qkm.4 for ; Mon, 29 Apr 2019 20:10:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=BJo+tsGYz1UiYdilzcTz+d7vEhUJ+jp/yMih2keKOkE=; b=Lt+vd+hO4XVo8ppOcrxWDOaD9FA42OxUrk9eZCwRUH4GvRMHFyMzX4JIsIKS7CHfYA 0kUuD8ZKP3CVmQsVtYgZ8GetWTgOPFylfvCwhoT2bgbSRMlDJJlC9Ib0rcWVvKh75UNQ MEKk3sky6k2lplRih/V/iVplFtDmsT8wqVZKVcwlsc6CzjwfE3mc3n7PIQSUTOHDDWf1 BfA4lnQuGJG7BgNuZlQ9AV26kE5QZapRBtU7MxuO/Vd28MM39EM2FB551uT7y3ZsYV1Q 1qqzdJny/KDRhCtFA3ITgJ/SJvSKOogqli70ZnlZUCm4lNEREKQR+/5sTWr5Ow/3P1Im OZbw== X-Gm-Message-State: APjAAAWchtPOacq0ZPaTGr2pn10vNgqcuLlfRlW9Fqwifa75SY1EKIwL vg+HhLNyNT2URe5XwjtFGYMx8EniAAlNROVvcDSoXIqeGLsKy7SHOuCuQ85ycaC+bXaYfBuePco Ch10VxJx6crOYgmWiYndILokT9HTuFwW26crQyfHyIKKOFLI/s6/Uasql9HLn6cM= X-Received: by 2002:a37:74c5:: with SMTP id p188mr27077876qkc.26.1556593846399; Mon, 29 Apr 2019 20:10:46 -0700 (PDT) X-Google-Smtp-Source: APXvYqxJbvv36Tk/flcmEzKjdWTpLjXcWDV07qaA16Kaf+jpv3DI2Cfos7N/Phiy/90QJVlf/k+I X-Received: by 2002:a37:74c5:: with SMTP id p188mr27077828qkc.26.1556593845171; Mon, 29 Apr 2019 20:10:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1556593845; cv=none; d=google.com; s=arc-20160816; b=IltXu5u0KXIB0eaY6OEzEYL6biWiUwytLPLCuE25IShAa+ui7GZJk4TTVN1LnjRBEG CQnhvGobXZ49q97/Q6jFFqpVfLb+OaAt/MPoExW+u0r3vX82U+DkxWUkIFnIzUPR9whn lakkPuql+56mmnxBkA4gaxf4oYz8MhCa52B+e3nzVmtyJG52FAOSLPLPvNwPB1QITwNd RDkrh5fCKLSF9GzC7h7FvSqCzD7yuPJ6UTuAoB8DDDAiGWLRl0igbeY7sx1bJ0mh27Uz 4NoNwyFMeB4lkyQbUwgRfKO+aj4yo25A08LGJPWI6zoS0+pC1yQYuf9BC1SNVWV3rQJl QUuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=BJo+tsGYz1UiYdilzcTz+d7vEhUJ+jp/yMih2keKOkE=; b=SimeB09ghUcqXQitUUAKnla36FwVRC5XcPidi/Tv42STJtsqnIkJUFNU5hVwJJaI1p Fonu7GqVpWGU3v/zhJ8fFfhsdNdk/OtlsBK5GtEPBoqKVRhoGgwaUezyayWWunNC16YN gFGN+pFRD1sJ0Pj6/5mWK8dVIEnGixsTaqFq/FxtQB+DVUj1AZC/2IK1mI/bgeic3tuG XMwi+6XzQOVF9LHITfbeTJiU9qe/zcOIB/2fM1K83nxgkV9Pq8vDNgIoN+7uExEs8Qu7 RkEZpaw29DkWeclMKtJZi+VzjtbQ6z1XnCatwLdmyOOmbl+xMFxQvwGrRWF4K37RsC+m ICCg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=4Na3wF3I; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com. [66.111.4.221]) by mx.google.com with ESMTPS id t24si3649556qte.96.2019.04.29.20.10.45 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Apr 2019 20:10:45 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) client-ip=66.111.4.221; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=4Na3wF3I; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.221 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailnew.nyi.internal (Postfix) with ESMTP id BF1FE9A40; Mon, 29 Apr 2019 23:10:44 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 29 Apr 2019 23:10:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=BJo+tsGYz1UiYdilzcTz+d7vEhUJ+jp/yMih2keKOkE=; b=4Na3wF3I x24V1uXx6GIXq0r+bQnjj+1SJ6YVMI1Ebj8HL9tPpVhag1zyRctWPePGEkZkDb4v T7JhLPqcmvcj9VujzI05sNPVwfXS0vPhfBS/txTRihZVWGrdwTWDzsjO7tbNbct2 3ZiYs4yFlIkuKixNaDXTwotfl3aRZjWGc2rDrpHWUu+BnJojeC1RYL+JixYcJUM6 C+JXs/TIyu6EyE4NieFwVUhgVzxXU7lRlrUVSF4AVEzYyMrFUl3DRRs5ubXilPyO Y35PGXi6rcuDrW/6tgzRfGh49RJelBQUWyRdHTfdrpcjbbL9uIz8RkLCWtYrNkDZ 1TNDxczMcD8R/w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrieefgdeilecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvuddrgeegrddvfedtrddukeeknecurfgrrhgrmhepmhgrihhlfhhrohhmpehtohgs ihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpeef X-ME-Proxy: Received: from eros.localdomain (ppp121-44-230-188.bras2.syd2.internode.on.net [121.44.230.188]) by mail.messagingengine.com (Postfix) with ESMTPA id EC48F103C8; Mon, 29 Apr 2019 23:10:36 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , Jonathan Corbet , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v4 15/15] dcache: Add CONFIG_DCACHE_SMO Date: Tue, 30 Apr 2019 13:07:46 +1000 Message-Id: <20190430030746.26102-16-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190430030746.26102-1-tobin@kernel.org> References: <20190430030746.26102-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP In an attempt to make the SMO patchset as non-invasive as possible add a config option CONFIG_DCACHE_SMO (under "Memory Management options") for enabling SMO for the DCACHE. Whithout this option dcache constructor is used but no other code is built in, with this option enabled slab mobility is enabled and the isolate/migrate functions are built in. Add CONFIG_DCACHE_SMO to guard the partial shrinking of the dcache via Slab Movable Objects infrastructure. Signed-off-by: Tobin C. Harding --- fs/dcache.c | 4 ++++ mm/Kconfig | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/fs/dcache.c b/fs/dcache.c index 3f9daba1cc78..9edce104613b 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -3068,6 +3068,7 @@ void d_tmpfile(struct dentry *dentry, struct inode *inode) } EXPORT_SYMBOL(d_tmpfile); +#ifdef CONFIG_DCACHE_SMO /* * d_isolate() - Dentry isolation callback function. * @s: The dentry cache. @@ -3140,6 +3141,7 @@ static void d_partial_shrink(struct kmem_cache *s, void **_unused, int __unused, kfree(private); } +#endif /* CONFIG_DCACHE_SMO */ static __initdata unsigned long dhash_entries; static int __init set_dhash_entries(char *str) @@ -3186,7 +3188,9 @@ static void __init dcache_init(void) sizeof_field(struct dentry, d_iname), dcache_ctor); +#ifdef CONFIG_DCACHE_SMO kmem_cache_setup_mobility(dentry_cache, d_isolate, d_partial_shrink); +#endif /* Hash may have been set up in dcache_init_early */ if (!hashdist) diff --git a/mm/Kconfig b/mm/Kconfig index 47040d939f3b..92fc27ad3472 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -265,6 +265,13 @@ config SMO_NODE help On NUMA systems enable moving objects to and from a specified node. +config DCACHE_SMO + bool "Enable Slab Movable Objects for the dcache" + depends on SLUB + help + Under memory pressure we can try to free dentry slab cache objects from + the partial slab list if this is enabled. + config PHYS_ADDR_T_64BIT def_bool 64BIT