From patchwork Wed Apr 3 04:21:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tobin C. Harding" X-Patchwork-Id: 10882757 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3170C17E1 for ; Wed, 3 Apr 2019 04:23:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1A9AC287A6 for ; Wed, 3 Apr 2019 04:23:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0DF9D289BF; Wed, 3 Apr 2019 04:23:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7EA79287A6 for ; Wed, 3 Apr 2019 04:23:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 644CB6B026D; Wed, 3 Apr 2019 00:23:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5F5AF6B027A; Wed, 3 Apr 2019 00:23:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4BDF56B027B; Wed, 3 Apr 2019 00:23:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by kanga.kvack.org (Postfix) with ESMTP id 2EDBA6B026D for ; Wed, 3 Apr 2019 00:23:54 -0400 (EDT) Received: by mail-qk1-f200.google.com with SMTP id 23so13538792qkl.16 for ; Tue, 02 Apr 2019 21:23:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=uAywuLhgM6+Szq4etbWPzwmP073FVonmktwU8/xEONg=; b=aMhMlIJGbF4QJ3XeSWpI7RPRl8AHO0cvNuM1KebXT12u66abNyLf4X+6f44L3RJKjY B6Y/iFYvvd2mpFvoGfCgHrmuSSOLye97Z4hB69a42Fyd3mOU0Ve3qpFqpCnQoTsiqruE +wCoEXCkYfKpgWOORAIVcHknaEtEb7UfrSsCkFPEO/iE1B9SVr96aC28lM7fCprVYSdS DUzaF+z6eIPP425snBsu0MwK0wBCjT9XyXjWpVL+AfqLvCj4a+yJzWHd9lMeAweOZkC3 9YXseuwVnueXXnq4KU1eOD2VMlhPVQA/ogAgeVM1PxDsZmjlAc+u8xAJB43dNmRqiUZx GAdA== X-Gm-Message-State: APjAAAVRTXkqs+lagCrEIGRhtBg1ii9P99OqisgeUsawlVoSFyByCL/n Qv0QGHLMwK29EtD5PeqeZQmyrZy4lRf5kfL2T3+2a6YNZmWWdnpznsfs8aid9VzHFHsgpLDyvbi ApAzMGQFoZEqAeXxW61F3fuFFIqw7L8o+dxln97JAlohx4MT2FDo85QDED+L+YSM= X-Received: by 2002:a05:620a:132b:: with SMTP id p11mr93947qkj.279.1554265433938; Tue, 02 Apr 2019 21:23:53 -0700 (PDT) X-Google-Smtp-Source: APXvYqzQn4UMtrPQ0Om/0T/QiDecaolJ1rWFJ39GPw76cJ4OEF0UfbvVKjZv7BJSiyFxkTPSSeV3 X-Received: by 2002:a05:620a:132b:: with SMTP id p11mr93899qkj.279.1554265432875; Tue, 02 Apr 2019 21:23:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554265432; cv=none; d=google.com; s=arc-20160816; b=uIkEHvvb4mmgTiQnSPBDFSBqOUAk3WuhhbX10BGkZa8dbK/7Mcui9/rjJrlh8Orx2v FPAtHOYLi/QBanr0TciMFGGWQaS+q/UxCnvXsq/Czv6FQS6qhF00IYMAdU3M/50OiEWX IO15I67pmI9gOPCbiRqV2Yq1Hby49h0SABmFhAHqg7emUWCkp8h93i1Kt/pCX+Ax+EHI MSEXUpls+R74yesvHJPmBYyA37tIY7TiLuoC0ifM/p80wDzo+WdamLulaJZh8U4GEAqg VC3mMYq/2nMmupyHwI8xnVWU2eSjjeYaRYVLJES3lXwU8DYv/RP8tuHq2vE7omV6lSi6 bmKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature; bh=uAywuLhgM6+Szq4etbWPzwmP073FVonmktwU8/xEONg=; b=F8cvRrBapqE55PJ9lcscjaJF7QzMvEHINwvQRYsBzO84LG9HueZ+tAFohxOWtuEybh YB96EPNzb7LkppAN1eX3Tsqk2/lbrCHu1gu1AOl66wJSxyoQKL5CRRHCSILnAXSo+yiB risdMt0f6AB9Zgapy3+Py/onXopMPxxB92CFoO2CdZAU6vY7/cQrcqgrdpe7PYWGrNyB RahXJEthxWDp7zgAciP+5sDxog6zO2e68uOvsfMNqYYv/9uh7zEp+AtN3OUtLJu6Q3Ax AE/g4G9n13ltN9b/aZjG97tn/+KdhxAMrLztDs3VgCU1AzAeEkXqoH0UuoWPG5MlavXz 8NzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=xeXsXdNf; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com. [66.111.4.28]) by mx.google.com with ESMTPS id n64si2500264qte.106.2019.04.02.21.23.52 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Apr 2019 21:23:52 -0700 (PDT) Received-SPF: softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) client-ip=66.111.4.28; Authentication-Results: mx.google.com; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=xeXsXdNf; spf=softfail (google.com: domain of transitioning tobin@kernel.org does not designate 66.111.4.28 as permitted sender) smtp.mailfrom=tobin@kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 91FB421B84; Wed, 3 Apr 2019 00:23:52 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 00:23:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=uAywuLhgM6+Szq4etbWPzwmP073FVonmktwU8/xEONg=; b=xeXsXdNf P0XevguDVWe8R0wWT3qYLmuo5crZKYPd4pltu4WJxBU0f1oEsa362LSpgZ4CdNQZ InDlUyX3uK85y8/EYAAhVz5TMUtIHMgU5XXYa5LIjJPFILKnrfjMeJrdXL/dSU6Y pJlOZ3Mo836ihiKeOtTmLe+/CGXJQ8tmDK3FsS8LQUztBoDAC5+NA+JnSY/Zs4F4 o+IXwNfDR69U8p86UcIwY8UpCknhFqteRa5q7IUTo/MbwI//s1f3+TKLxZTEVbA8 HKFIpylGzC69vJxjlgNyf1x9aio4JulIFvN5CtZCFtfVq2Sk50jrULAcxMsV3Mx1 hz3eqk7G3QXNhg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtddugdektdculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhgggfestdekredtredttden ucfhrhhomhepfdfvohgsihhnucevrdcujfgrrhguihhnghdfuceothhosghinheskhgvrh hnvghlrdhorhhgqeenucfkphepuddvgedrudeiledrvdejrddvtdeknecurfgrrhgrmhep mhgrihhlfhhrohhmpehtohgsihhnsehkvghrnhgvlhdrohhrghenucevlhhushhtvghruf hiiigvpeek X-ME-Proxy: Received: from eros.localdomain (124-169-27-208.dyn.iinet.net.au [124.169.27.208]) by mail.messagingengine.com (Postfix) with ESMTPA id 9597A100E5; Wed, 3 Apr 2019 00:23:45 -0400 (EDT) From: "Tobin C. Harding" To: Andrew Morton Cc: "Tobin C. Harding" , Roman Gushchin , Alexander Viro , Christoph Hellwig , Pekka Enberg , David Rientjes , Joonsoo Kim , Christopher Lameter , Matthew Wilcox , Miklos Szeredi , Andreas Dilger , Waiman Long , Tycho Andersen , "Theodore Ts'o" , Andi Kleen , David Chinner , Nick Piggin , Rik van Riel , Hugh Dickins , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 09/14] xarray: Implement migration function for objects Date: Wed, 3 Apr 2019 15:21:22 +1100 Message-Id: <20190403042127.18755-10-tobin@kernel.org> X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190403042127.18755-1-tobin@kernel.org> References: <20190403042127.18755-1-tobin@kernel.org> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Implement functions to migrate objects. This is based on initial code by Matthew Wilcox and was modified to work with slab object migration. Cc: Matthew Wilcox Co-developed-by: Christoph Lameter Signed-off-by: Tobin C. Harding --- lib/radix-tree.c | 13 +++++++++++++ lib/xarray.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/lib/radix-tree.c b/lib/radix-tree.c index 14d51548bea6..9412c2853726 100644 --- a/lib/radix-tree.c +++ b/lib/radix-tree.c @@ -1613,6 +1613,17 @@ static int radix_tree_cpu_dead(unsigned int cpu) return 0; } +extern void xa_object_migrate(void *tree_node, int numa_node); + +static void radix_tree_migrate(struct kmem_cache *s, void **objects, int nr, + int node, void *private) +{ + int i; + + for (i = 0; i < nr; i++) + xa_object_migrate(objects[i], node); +} + void __init radix_tree_init(void) { int ret; @@ -1627,4 +1638,6 @@ void __init radix_tree_init(void) ret = cpuhp_setup_state_nocalls(CPUHP_RADIX_DEAD, "lib/radix:dead", NULL, radix_tree_cpu_dead); WARN_ON(ret < 0); + kmem_cache_setup_mobility(radix_tree_node_cachep, NULL, + radix_tree_migrate); } diff --git a/lib/xarray.c b/lib/xarray.c index 6be3acbb861f..6d2657f2e4cb 100644 --- a/lib/xarray.c +++ b/lib/xarray.c @@ -1971,6 +1971,52 @@ void xa_destroy(struct xarray *xa) } EXPORT_SYMBOL(xa_destroy); +void xa_object_migrate(struct xa_node *node, int numa_node) +{ + struct xarray *xa = READ_ONCE(node->array); + void __rcu **slot; + struct xa_node *new_node; + int i; + + /* Freed or not yet in tree then skip */ + if (!xa || xa == XA_RCU_FREE) + return; + + new_node = kmem_cache_alloc_node(radix_tree_node_cachep, + GFP_KERNEL, numa_node); + if (!new_node) + return; + + xa_lock_irq(xa); + + /* Check again..... */ + if (xa != node->array || !list_empty(&node->private_list)) { + node = new_node; + goto unlock; + } + + memcpy(new_node, node, sizeof(struct xa_node)); + + /* Move pointers to new node */ + INIT_LIST_HEAD(&new_node->private_list); + for (i = 0; i < XA_CHUNK_SIZE; i++) { + void *x = xa_entry_locked(xa, new_node, i); + + if (xa_is_node(x)) + rcu_assign_pointer(xa_to_node(x)->parent, new_node); + } + if (!new_node->parent) + slot = &xa->xa_head; + else + slot = &xa_parent_locked(xa, new_node)->slots[new_node->offset]; + rcu_assign_pointer(*slot, xa_mk_node(new_node)); + +unlock: + xa_unlock_irq(xa); + xa_node_free(node); + rcu_barrier(); +} + #ifdef XA_DEBUG void xa_dump_node(const struct xa_node *node) {