From patchwork Fri Mar 8 04:14:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10844167 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4708A922 for ; Fri, 8 Mar 2019 04:15:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 334732E18A for ; Fri, 8 Mar 2019 04:15:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 27DEA2E1B0; Fri, 8 Mar 2019 04:15:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9B73D2E18A for ; Fri, 8 Mar 2019 04:15:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4C68C8E000F; Thu, 7 Mar 2019 23:15:45 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 49D588E0002; Thu, 7 Mar 2019 23:15:45 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3663B8E000F; Thu, 7 Mar 2019 23:15:45 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id 066C08E0002 for ; Thu, 7 Mar 2019 23:15:45 -0500 (EST) Received: by mail-qk1-f199.google.com with SMTP id i66so14969163qke.21 for ; Thu, 07 Mar 2019 20:15:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=3ICf1F4yYMSB9kWf6DNRwgEAL3fOm46vEzPIjpQE3AA=; b=gNM2kvVD7gJTjYHlu5H7/Ae2xTbRgc4VXFH+GedR/FxWJS6+bxQHPAaQIst5Q20isI Tjvqt3WM/l7H6jW3QvzsRqhR8TqqcbN3kXloPQD65h5RXWaSAjGufuNpgSdji2jUtNEE ji/Ap/X/My9lhNrLSUrsOz4Jk19XYTqxTUf54WvuYNV2iS9e/I/Jj4UgR1HvoQ0gMWjF U9oSgzHvE5eLKcWY9PPcJBbt9XR0dY8ynxGYvzZqTRVgu1Q6QW/5M3z/6MujJLHIHDj0 jZEOVXinEPGxxzQDWP0OXbPWjLKuBKxr635bibYoAg+KZJbrBgFtnO2SUxwj8RwIsvhj 41hg== X-Gm-Message-State: APjAAAUQZTQCBGOBfGr2hGN7a4qIxzEeMcHSI/sY0HdSjsGriTw5Mq25 nwxwU9LjzcbmhZcOKLtqnNmpD6bcF28g7Y/Fy90ddZzehIO4l/kbmB3D5u+wlQBfUsscVSmFZ32 +azf6Rop/KMOA+9Q0JXX9j0yUNqyZZKXx/pqS6tUxv4bEnnA+ezN+Md+K1IZOORE= X-Received: by 2002:ae9:e913:: with SMTP id x19mr12699037qkf.45.1552018544788; Thu, 07 Mar 2019 20:15:44 -0800 (PST) X-Google-Smtp-Source: APXvYqxaAa676OUvZnP8ColV8X+YpWynk+qJd60PP9EPSwz41bpr5hcTsV3RZh09x5VilYh7+/x6 X-Received: by 2002:ae9:e913:: with SMTP id x19mr12699001qkf.45.1552018543689; Thu, 07 Mar 2019 20:15:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1552018543; cv=none; d=google.com; s=arc-20160816; b=go0sI4rr8+DIiCNijk72FvB9dXIefZAVs2qHqfnt+2tSDue+0m+yuSbnByEjySEXF2 uNYCD7uuWPo3/KQng+RmhSgcfIParwWtBJT7o8RYPPxo/+Ir3DcxeE8gAjESM0ECHh/w E/7cxJCqXbcRvLmNCDZT9Pjt3HKbJvSz6zbshsWuab7nvT6+jNz8l+r+z5gX1+aTwfy4 rLfrZzUrXIIer9qiuJkhzwXVMXwmA4muuwX/rO7cmKOYvd3M7UZOY23/HWnvqjgageCE yBGMN+vILgamSrR1vWnwGZzrWmK8FAHWydR5c4+hEMgMM+RV44pnvItMZSrJEyEQsoi5 sH6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=3ICf1F4yYMSB9kWf6DNRwgEAL3fOm46vEzPIjpQE3AA=; b=EKndcbjKvg5dkBrap41HoTr75/4VSBu3pXVyrMtnYz7HOucLYWiJrH80STxi+6vm3p mCwbkvVnDoYNjeLLOyjhUqlxb4pXhGxjP7iR7h6ddAKya+LnuBT11sBKY8iD7nExRz0o 905V6A/NGa7l/UZ7WunX5u+j2fLvDX7OjfH0AmWyEkLEHkoAa86CGv0jTaOiW7Oy+HvB H0S8MUe3kKV6XHt+qrvwJ1LMlKSsXf/mO0PhosxQjzLpik/Z3JVxYeLmZ/Gxytn7GKjE RIOjVsq4Lfg4l7VSOiTy+JWxR3QASAzMDmdi5Z+ufRckAyvK4ezM1yJX/ezmdEfZlGNt aFDQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=wnPbtlN6; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 64.147.123.25 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from wout2-smtp.messagingengine.com (wout2-smtp.messagingengine.com. [64.147.123.25]) by mx.google.com with ESMTPS id p6si79931qkk.40.2019.03.07.20.15.43 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Mar 2019 20:15:43 -0800 (PST) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 64.147.123.25 as permitted sender) client-ip=64.147.123.25; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=wnPbtlN6; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 64.147.123.25 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 3F395173F; Thu, 7 Mar 2019 23:15:42 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Thu, 07 Mar 2019 23:15:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=3ICf1F4yYMSB9kWf6DNRwgEAL3fOm46vEzPIjpQE3AA=; b=wnPbtlN6 RC57mehAdbkgLwnR/WvJ/TT8udSoYHVbvFCmZkCjkSaSMoUAm5LlwMU312bg57+H 6xwUIAkxleRGO/DO1Mo2+EvPzombVF6OqLWYKPFTldgVN9nDbPCpJQGa1MDwpLhB DFyag4Efz62jiOXOvKDvJIfFE5Z2yGLheTL5xI8RPuZu8/X0Eb2m616FqlsJPjR1 dEtwyMJrztDmNqW4SbHfw665vzC4m7Vb3PAFpX+MOUXgQoPEMHhtaOupOqYaqtqP q9BdKJ08/wTkvvix56kpnZDHp8l9wijvkwFdnG+kSzHMuZqGz1byU5qfGsrzT+gy p4CKCE9JbR+oag== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedutddrfeelgdeifecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepfdfvohgsihhn ucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrhhnvghlrdhorhhgqeenucfkph epuddvgedrudeiledrhedrudehkeenucfrrghrrghmpehmrghilhhfrhhomhepthhosghi nheskhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgepuddu X-ME-Proxy: Received: from eros.localdomain (124-169-5-158.dyn.iinet.net.au [124.169.5.158]) by mail.messagingengine.com (Postfix) with ESMTPA id EB429E4362; Thu, 7 Mar 2019 23:15:38 -0500 (EST) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Christopher Lameter , Pekka Enberg , Matthew Wilcox , Tycho Andersen , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC 12/15] xarray: Implement migration function for objects Date: Fri, 8 Mar 2019 15:14:23 +1100 Message-Id: <20190308041426.16654-13-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190308041426.16654-1-tobin@kernel.org> References: <20190308041426.16654-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Implement functions to migrate objects. This is based on initial code by Matthew Wilcox and was modified to work with slab object migration. Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- lib/radix-tree.c | 13 +++++++++++++ lib/xarray.c | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 14d51548bea6..9412c2853726 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -1613,6 +1613,17 @@ static int radix_tree_cpu_dead(unsigned int cpu) return 0; } +extern void xa_object_migrate(void *tree_node, int numa_node); + +static void radix_tree_migrate(struct kmem_cache *s, void **objects, int nr, + int node, void *private) +{ + int i; + + for (i = 0; i < nr; i++) + xa_object_migrate(objects[i], node); +} + void __init radix_tree_init(void) { int ret; @@ -1627,4 +1638,6 @@ void __init radix_tree_init(void) ret = cpuhp_setup_state_nocalls(CPUHP_RADIX_DEAD, "lib/radix:dead", NULL, radix_tree_cpu_dead); WARN_ON(ret < 0); + kmem_cache_setup_mobility(radix_tree_node_cachep, NULL, + radix_tree_migrate); } diff --git a/lib/xarray.c b/lib/xarray.c index 81c3171ddde9..4f6f17c87769 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1950,6 +1950,50 @@ void xa_destroy(struct xarray *xa) } EXPORT_SYMBOL(xa_destroy); +void xa_object_migrate(struct xa_node *node, int numa_node) +{ + struct xarray *xa = READ_ONCE(node->array); + void __rcu **slot; + struct xa_node *new_node; + int i; + + /* Freed or not yet in tree then skip */ + if (!xa || xa == XA_FREE_MARK) + return; + + new_node = kmem_cache_alloc_node(radix_tree_node_cachep, + GFP_KERNEL, numa_node); + + xa_lock_irq(xa); + + /* Check again..... */ + if (xa != node->array || !list_empty(&node->private_list)) { + node = new_node; + goto unlock; + } + + memcpy(new_node, node, sizeof(struct xa_node)); + + /* Move pointers to new node */ + INIT_LIST_HEAD(&new_node->private_list); + for (i = 0; i < XA_CHUNK_SIZE; i++) { + void *x = xa_entry_locked(xa, new_node, i); + + if (xa_is_node(x)) + rcu_assign_pointer(xa_to_node(x)->parent, new_node); + } + if (!new_node->parent) + slot = &xa->xa_head; + else + slot = &xa_parent_locked(xa, new_node)->slots[new_node->offset]; + rcu_assign_pointer(*slot, xa_mk_node(new_node)); + +unlock: + xa_unlock_irq(xa); + xa_node_free(node); + rcu_barrier(); +} + #ifdef XA_DEBUG void xa_dump_node(const struct xa_node *node) {