From patchwork Tue Nov 13 09:12:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wei Yang X-Patchwork-Id: 10680097 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3C29E13BB for ; Tue, 13 Nov 2018 09:12:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 273B429C07 for ; Tue, 13 Nov 2018 09:12:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 185B829C09; Tue, 13 Nov 2018 09:12:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9ECF429C07 for ; Tue, 13 Nov 2018 09:12:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A62296B0007; Tue, 13 Nov 2018 04:12:50 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A104A6B0008; Tue, 13 Nov 2018 04:12:50 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9275C6B000A; Tue, 13 Nov 2018 04:12:50 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by kanga.kvack.org (Postfix) with ESMTP id 541626B0007 for ; Tue, 13 Nov 2018 04:12:50 -0500 (EST) Received: by mail-pf1-f200.google.com with SMTP id e89so1867093pfb.17 for ; Tue, 13 Nov 2018 01:12:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=kjr2sDW4EZNAp5P1/YPoT2Nfrrhh6PvtASGQ5IjSD/k=; b=f/87fjPYhKN8Lj9pyd9a92Fu+gvWfA3wdPtmiYA5koDE/1ovLEzRarRqq2jQXkr3WT lJOuHu0h5kT7WYAhekIYIKT0QnRQ7mcp6P5kwp24EEhTLbw9FTDvBK8BheS4qVTtclTf o1iSDaLaVvt50BOE+n+98IGHMic1jZuVlAmK4n7zk5EvOSkkdQDJDRBgqbt6skgGmbfC HEccXuNwk/0UsQQ2+uIem/BihXk3jNpysW6VU1JsaR3FLPsIYOpxEUinKqIwPu3AOKp2 3Z6orO4oR0TW6pvOoMt7oxPju3EO2Vd1zlOZIBv+uNNNpRkMWUoxTa9BMVw4O8H+YaL5 lT+g== X-Gm-Message-State: AGRZ1gJdekP/YGsDMy9uIBoxiFRv9ZC9/QfVfqQuBm59n8NL0At7uS/H apE13CCnZ6jMUd7686uVqfhPFFIsUkZfUpaknu2JU5Kp1L3ZluJbnP2FZEBBTW9eNCqi+W3mSv6 U6qwe5cgS7CBi1k49i+FuUnV3H03qk03zQ35gJLRLoOLljr2W5QGHPxY9eLWUH5N5EpXN96ZOLZ ctBTvGd1gk1w2lx+dFKqlVgJhhVoioIC9HnLnjw4gW7vINqpxvA5jVAdze4nsCmb4h7e5c4Z8lq QyzM5W/YTn18GRa9hke02LlBr8Rp1TYsrjkHb7YrWyeTd2nwDmgpBC9G63QMkjpzJ4zMWwgJv2k bm+4YBCsRaLJXek36ce6DIJYJxWY3x49jWCJRUcsR+GheRpVV/7AVPxGf9ReV+wPdGU1yULxayw j X-Received: by 2002:a62:5dd1:: with SMTP id n78mr2880126pfj.58.1542100369902; Tue, 13 Nov 2018 01:12:49 -0800 (PST) X-Received: by 2002:a62:5dd1:: with SMTP id n78mr2880084pfj.58.1542100369035; Tue, 13 Nov 2018 01:12:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1542100369; cv=none; d=google.com; s=arc-20160816; b=B/cUQqo3nHMhX/A2h4pj8iiOdd7oBjqE4ELI8sjt8oiBus2OAo0lwkdfaCpYvbQdCE gi6nSXXVZm73CXkRb6BkCy6vuF9eCEdxX6gC9EV+c8TDYkeTgsZyA+sW4mojabc1yj9/ rD4Ir7KLtsv8fcVM4fdCYtFmwbKDtPfh6XEG416aUP2ck64sxB6PvgmsY5oZKyKtbEyK uaSi6YQ7VLIxtnAH2Rj5MIo5lrV8gFfT+ErDZg0j4lWe8fmh7276hSaP1f/Myt4Fwn6N 3lO59SbtEHu5uLnhyUE64mvYtG9b2rU78z84q3xkmKzmZTOs90bboFnxm57NLQre2M26 lxNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kjr2sDW4EZNAp5P1/YPoT2Nfrrhh6PvtASGQ5IjSD/k=; b=hPowr+MdOHxCPNEf601brGNUZp4QCgR4srn+kvMXhqPjMxjzf5DF4Yaybe1pOQoPF/ OzWDKjdroDocAojbl69Oe0dloeNkYqdJF3HfXwHmjbCoArhxoqpqve1jJ+FLZ4sZU/Cd bL44iTS/cU8E84QZjc46c+dTU4JcZ1j/6lqJz1VnwTQUjfLJt+AlTimVUjwwwjq8zAsA 8J0lz06GpcXODwOi/TgctPF6jcTzPhDDIwXNf9iXeVf1t7SFNnzkMp6VzaN24ktwPIBm p8lGvRxr0D4QfmEV+sJ6dRPhd3YaxFN98/AZgig1wYyjlFiT4R1BoBGicMCAtr0ToBpY vgvw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=QZD0Ib15; spf=pass (google.com: domain of richard.weiyang@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=richard.weiyang@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id y23sor7542595pga.35.2018.11.13.01.12.48 for (Google Transport Security); Tue, 13 Nov 2018 01:12:49 -0800 (PST) Received-SPF: pass (google.com: domain of richard.weiyang@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=QZD0Ib15; spf=pass (google.com: domain of richard.weiyang@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=richard.weiyang@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=kjr2sDW4EZNAp5P1/YPoT2Nfrrhh6PvtASGQ5IjSD/k=; b=QZD0Ib15Gcq8o4ow3JWbpUTcgzMkxuOahxnU3s9BbG6dPy1PMCwRwy3yx9/7V566UE vMBKAd2DBAM1++8F1C8UjkWD6MG2NpEeqm8sy4Wl9UUZnF2QVNpTJ7sj/vm8Aoxa6U5z pwG+byVmi/x2WObGsLZLFurQrFVZ+FG0savY3gJqDiEIwaG/hAZ4m2MnEoLc8ZxXsp1P TFS1YBr9SLcHLrhGMTzy/lDG79TXd2JreYTKb5Z5ewOwVvXLsltZa0WrTIJjnHCwXoJM H3cIDFfD/eQ6dSUuUhEsaRD15XO6tW6wx7Mg3m+7EImo3Hnnnb7LrmyC/J0WhuWghsdS TnXw== X-Google-Smtp-Source: AJdET5cgedu9vIDXo0KZvExjf6wjbZ92YrHuqKVkM4cmwBIGMMoyR/jc8ofJvNXFx5RS8QIGGkidYg== X-Received: by 2002:a63:e302:: with SMTP id f2mr4054621pgh.320.1542100368386; Tue, 13 Nov 2018 01:12:48 -0800 (PST) Received: from localhost ([185.92.221.13]) by smtp.gmail.com with ESMTPSA id z30-v6sm9726350pfg.41.2018.11.13.01.12.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 13 Nov 2018 01:12:47 -0800 (PST) From: Wei Yang To: cl@linux.com, penberg@kernel.org, akpm@linux-foundation.org Cc: linux-mm@kvack.org, Wei Yang Subject: [PATCH v2] mm/slub: skip node in case there is no slab to acquire Date: Tue, 13 Nov 2018 17:12:40 +0800 Message-Id: <20181113091240.23308-1-richard.weiyang@gmail.com> X-Mailer: git-send-email 2.15.1 In-Reply-To: <20181108011204.9491-1-richard.weiyang@gmail.com> References: <20181108011204.9491-1-richard.weiyang@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Current slub has three layers: * cpu_slab * percpu_partial * per node partial list Slub allocator tries to get an object from top to bottom. When it can't get an object from the upper two layers, it will search the per node partial list. The is done in get_partial(). The abstraction of get_partial() may looks like this: get_partial() get_partial_node() get_any_partial() for_each_zone_zonelist() The idea behind this is: it first try a local node, then try other nodes if caller doesn't specify a node. When we look one step deeper in get_any_partial(), it tries to get a proper node by for_each_zone_zonelist(), which iterates on the node_zonelists. This behavior would introduce some redundant check on the same node. Because: * the local node is already checked in get_partial_node() * one node may have several zones on node_zonelists We could reduce these redundant check by providing a nodemask during node_zonelists iteration. * clear the local node which is already checked in get_partial_node() * clear a node if we can't get an object from it. This patch replaces for_each_zone_zonelist() with for_each_zone_zonelist_nodemask() to skip the node which fails to acquire an object. Signed-off-by: Wei Yang --- v2: rewrite the changelog and add a comment based on Andrew's comment --- mm/slub.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index e3629cd7aff1..e3db5cd52507 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -1873,7 +1873,7 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n, * Get a page from somewhere. Search in increasing NUMA distances. */ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, - struct kmem_cache_cpu *c) + struct kmem_cache_cpu *c, int except) { #ifdef CONFIG_NUMA struct zonelist *zonelist; @@ -1882,6 +1882,9 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, enum zone_type high_zoneidx = gfp_zone(flags); void *object; unsigned int cpuset_mems_cookie; + nodemask_t nmask = node_states[N_MEMORY]; + + node_clear(except, nmask); /* * The defrag ratio allows a configuration of the tradeoffs between @@ -1908,7 +1911,8 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, do { cpuset_mems_cookie = read_mems_allowed_begin(); zonelist = node_zonelist(mempolicy_slab_node(), flags); - for_each_zone_zonelist(zone, z, zonelist, high_zoneidx) { + for_each_zone_zonelist_nodemask(zone, z, zonelist, + high_zoneidx, &nmask) { struct kmem_cache_node *n; n = get_node(s, zone_to_nid(zone)); @@ -1926,6 +1930,11 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, */ return object; } + /* + * Fail to get object from this node, + * clear this to skip this node + */ + node_clear(zone_to_nid(zone), nmask); } } } while (read_mems_allowed_retry(cpuset_mems_cookie)); @@ -1951,7 +1960,7 @@ static void *get_partial(struct kmem_cache *s, gfp_t flags, int node, if (object || node != NUMA_NO_NODE) return object; - return get_any_partial(s, flags, c); + return get_any_partial(s, flags, c, searchnode); } #ifdef CONFIG_PREEMPT