From patchwork Thu Dec 20 19:21:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Christoph Lameter (Ampere)" X-Patchwork-Id: 10739427 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4FEA4746 for ; Thu, 20 Dec 2018 19:22:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3E8B328CA1 for ; Thu, 20 Dec 2018 19:22:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 32D2128D22; Thu, 20 Dec 2018 19:22:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CC35728CA1 for ; Thu, 20 Dec 2018 19:21:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E13BC8E000D; Thu, 20 Dec 2018 14:21:57 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DEC688E0001; Thu, 20 Dec 2018 14:21:57 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C64EE8E000D; Thu, 20 Dec 2018 14:21:57 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 9A4CB8E0001 for ; Thu, 20 Dec 2018 14:21:57 -0500 (EST) Received: by mail-qt1-f199.google.com with SMTP id t18so2992152qtj.3 for ; Thu, 20 Dec 2018 11:21:57 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:subject:references:mime-version :content-disposition:feedback-id; bh=WG4AB0pljQcpBmeuz1B8F5G9pA0mj9v/xq+gyFWGAVI=; b=U/pcIHomXB+TEO0956aXHnspCAUD4ZJ/7xOVwagLAyDJMif0BtB3LCLKTfWWu+AQZu lgw4fy/AP8WuyhZRRYayKSB7P4YdH/yrfiQxSEHIyPwitTHpz7h4G/uuAcFblT9EjLMK xu8vmtK7oerHnhPgjPRdrlbB6mBf2ie2fbYF11za2Fz01JdQgEFeqZCKURBEePdV7DiQ Yg6CYQFdTbJOsm9a5anER+Nriq2MHa9CRNEYohMZohbA0GaJJ+VCUeoZihSwGyt0TQ6l OYxYerLhmnFlPNS9/LZN469it9bNUGXOOJmQUxRdF9juRKH6FA+5nEUlBGBS//PBL5UU wsWQ== X-Gm-Message-State: AA+aEWbu2aFFNF1GMjYjnkToakkGl1lYthiqA8jHXb3Bj6Y5Rp+YKdeJ +0RHIhH1cXEyppUTIZwt8+byJckcJId+5NbReQf54h7kNn+FIFN6K0T2yjEEwlkcN96frEE3s8i OMVYteLv6A7BQs8NXWbaKc+yJzMCq6lQUbIS60j7RrqaTt4cZYYe2yMxVrJW/NAc= X-Received: by 2002:ac8:6898:: with SMTP id m24mr28227557qtq.57.1545333717431; Thu, 20 Dec 2018 11:21:57 -0800 (PST) X-Google-Smtp-Source: AFSGD/XfXJmiNZGtlSKtxLyBpfhQWuaUTnGIsWzGVzL8xT1FZajbex5IbmYvPsd+aATC5YJjDqLK X-Received: by 2002:ac8:6898:: with SMTP id m24mr28227525qtq.57.1545333716797; Thu, 20 Dec 2018 11:21:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545333716; cv=none; d=google.com; s=arc-20160816; b=dkforKuo6JVFwCdbfubDgI1av/Sl8a4txo11YoExFFa4Adbra6boZcDQrdImXPK6b+ c46XBKNRTtffW5ELVThGDwLKvtXuLr1mqwACPt5OQG886HqCoMNJ5+f/qkidKga1MmMU XvZLXqTXXrBvgmVFMJitk1moJYkU3lVjoSoJhvqT+49oxx22CZlOAmj/qtteR5XDR662 iPSEvmxwrTmxuSrcGcO0Akqsn1yIarXKY6oEDRWMt2EGliUSf/b74qxG8h3l6MDNO5Gd ZougGsuO5AtGj4dPQ4Tk1sldYlZX+/nPTG8EP6EE1NYiUXb1fnIDhy4wB+x5OCOf/8uQ p3ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-disposition:mime-version:references:subject:cc :cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:to:from:date:user-agent:message-id :dkim-signature; bh=WG4AB0pljQcpBmeuz1B8F5G9pA0mj9v/xq+gyFWGAVI=; b=krRIRXH7PWvW5sAQvsa1dfzenjjG2i2EvoX1VRwU7KxYYzFtt3nprgwvIE/vrxWNJp HkTeuZ7T8Ozm9H5UuIa6pPAudk0KDsk47PSY964oDi6k2umdIlA9aV1RgSN/4gjh+lTy YgaY5S+dYvpO4/nntXEFFTGSxstzMMfvMyDpC8qAoNHmDrZeeijtMMEUuPRb01RxqrH8 c5PTFfLfT9ANOEb6Z3p9oe2594oAY8dJngiGipK0ovn0UgyNc56Bb7vAFRtebEcm3aIU Y12yNlvulehjqyD0lyrLEd6iZnfrmBfG/pjoiIInyDv+rMFJSFXk6rRqnQyX4owODevO Wc1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=TfSWvzq+; spf=pass (google.com: domain of 01000167cd113509-2c85d8a3-8e0e-4cb6-a745-88733e448471-000000@amazonses.com designates 54.240.9.30 as permitted sender) smtp.mailfrom=01000167cd113509-2c85d8a3-8e0e-4cb6-a745-88733e448471-000000@amazonses.com Received: from a9-30.smtp-out.amazonses.com (a9-30.smtp-out.amazonses.com. [54.240.9.30]) by mx.google.com with ESMTPS id o5si1091733qkc.148.2018.12.20.11.21.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 20 Dec 2018 11:21:56 -0800 (PST) Received-SPF: pass (google.com: domain of 01000167cd113509-2c85d8a3-8e0e-4cb6-a745-88733e448471-000000@amazonses.com designates 54.240.9.30 as permitted sender) client-ip=54.240.9.30; Authentication-Results: mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=TfSWvzq+; spf=pass (google.com: domain of 01000167cd113509-2c85d8a3-8e0e-4cb6-a745-88733e448471-000000@amazonses.com designates 54.240.9.30 as permitted sender) smtp.mailfrom=01000167cd113509-2c85d8a3-8e0e-4cb6-a745-88733e448471-000000@amazonses.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1545333716; h=Message-Id:Date:From:To:Cc:Cc:Cc:CC:Cc:Cc:Cc:Cc:Cc:Cc:Cc:Subject:References:MIME-Version:Content-Type:Feedback-ID; bh=VdAFswNkZMzX1QnleI6nNoe36Hgyo5dlWLZMARfK7f8=; b=TfSWvzq+iWaub44mB9Mn8ltb+L1g6QW1S5kgNN/HUVsnK258K9k740hUbuDhEOMG bYTUyXxdL0Y3gmb2B+0S4/pIEqgRxEs5Qta6XL+73oW3WxvfSEyEN9EAbgWYzKYXFMv 5dYhndZ2/byo4T9TMaGK71a7AaJMFgPyIpndY2ts= Message-ID: <01000167cd113509-2c85d8a3-8e0e-4cb6-a745-88733e448471-000000@email.amazonses.com> User-Agent: quilt/0.65 Date: Thu, 20 Dec 2018 19:21:56 +0000 From: Christoph Lameter To: Matthew Wilcox Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: Pekka Enberg CC: akpm@linux-foundation.org Cc: Mel Gorman Cc: andi@firstfloor.org Cc: Rik van Riel Cc: Dave Chinner Cc: Christoph Hellwig Cc: Michal Hocko Cc: Mike Kravetz Subject: [RFC 1/7] slub: Replace ctor field with ops field in /sys/slab/* References: <20181220192145.023162076@linux.com> MIME-Version: 1.0 Content-Disposition: inline; filename=ctor_to_ops X-SES-Outgoing: 2018.12.20-54.240.9.30 Feedback-ID: 1.us-east-1.fQZZZ0Xtj2+TD7V5apTT/NrT6QKuPgzCT/IC7XYgDKI=:AmazonSES X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Create an ops field in /sys/slab/*/ops to contain all the callback operations defined for a slab cache. This will be used to display the additional callbacks that will be defined soon to enable defragmentation. Display the existing ctor callback in the ops fields contents. Signed-off-by: Christoph Lameter --- mm/slub.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) Index: linux/mm/slub.c =================================================================== --- linux.orig/mm/slub.c +++ linux/mm/slub.c @@ -4994,13 +4994,18 @@ static ssize_t cpu_partial_store(struct } SLAB_ATTR(cpu_partial); -static ssize_t ctor_show(struct kmem_cache *s, char *buf) +static ssize_t ops_show(struct kmem_cache *s, char *buf) { + int x = 0; + if (!s->ctor) return 0; - return sprintf(buf, "%pS\n", s->ctor); + + if (s->ctor) + x += sprintf(buf + x, "ctor : %pS\n", s->ctor); + return x; } -SLAB_ATTR_RO(ctor); +SLAB_ATTR_RO(ops); static ssize_t aliases_show(struct kmem_cache *s, char *buf) { @@ -5413,7 +5418,7 @@ static struct attribute *slab_attrs[] = &objects_partial_attr.attr, &partial_attr.attr, &cpu_slabs_attr.attr, - &ctor_attr.attr, + &ops_attr.attr, &aliases_attr.attr, &align_attr.attr, &hwcache_align_attr.attr, From patchwork Thu Dec 20 19:21:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Christoph Lameter (Ampere)" X-Patchwork-Id: 10739429 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 727A714E5 for ; Thu, 20 Dec 2018 19:22:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 60E7728CA1 for ; Thu, 20 Dec 2018 19:22:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5535328D22; Thu, 20 Dec 2018 19:22:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 70D4728CA1 for ; Thu, 20 Dec 2018 19:22:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1BF278E000E; Thu, 20 Dec 2018 14:21:59 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 11E5D8E0001; Thu, 20 Dec 2018 14:21:59 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00D2C8E000E; Thu, 20 Dec 2018 14:21:58 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by kanga.kvack.org (Postfix) with ESMTP id BD8268E0001 for ; Thu, 20 Dec 2018 14:21:58 -0500 (EST) Received: by mail-qk1-f198.google.com with SMTP id d196so2914695qkb.6 for ; Thu, 20 Dec 2018 11:21:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:subject:references:mime-version :content-disposition:feedback-id; bh=Ivmf8KcU0nLZ7jKiUwDMxrV5SnY1/n4WH4SoklosWW0=; b=p494fjCXRBtmikzfY3fSg8Y97E3Tt5xTe/oV+Qa/3zJL48ikRl2afKvKtSf0JZYCb9 M4DI1Cn2UxJTvu3SHMxIgHqKYyg7M4HPp7v5wXe/nbya6NfkIAYKMzkkS1R3NuqLAaFe UkTM+slRZkfr8193h6d/Sl/BnskaWrW4yIUVq1q4Wn5rZfQt9doc3OsuexpwRui80vxX XxtzArTIBplA5xvqkXpAoP8rI/c0JI45MlFlLbSEsrfSYaAnPX0OFuX4rWZON62DB2jX MR+Fu+CUaN//ki5LBYs95+s1FGZZgReKrNk8bfbSDO0vdh4w8gpZKdCgPNpo0jV0qNZ+ xzyg== X-Gm-Message-State: AA+aEWb0trggFIxrxdPadFKQwbfNA/jrej/KrEXnjMw8qcNX1oRZugwO /Bhs4erXvGyRvqcM+AdTVoG3UU2nYD+v1ujFyVUqtOxUHws+Tvpoejo10UZyxH0hy84oaDx+0Re lU8D/2vfTdXq453PZTbO5fTQJ8AYmuEW1DTeVL2M49HkqJhsFvlnYrBMZqObq0AU= X-Received: by 2002:ad4:42d1:: with SMTP id f17mr27312337qvr.59.1545333718517; Thu, 20 Dec 2018 11:21:58 -0800 (PST) X-Google-Smtp-Source: AFSGD/U6NTLTduKya/Jaj1UeeGKOzS/cg6mHz7sndzifOXZ+tyk+s21gMNcW0+MrrGhFiw40jr25 X-Received: by 2002:ad4:42d1:: with SMTP id f17mr27312314qvr.59.1545333718159; Thu, 20 Dec 2018 11:21:58 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545333718; cv=none; d=google.com; s=arc-20160816; b=K6C2LCMlLa+tKqJjyssKFOl6ThH36Y6n+p4d/w1U7DOzrecCMoPjKbUYkwdScFNz2V E/NvO9RJFqofKclfj3ztGk807TrPp87agUJAVNChbcI6EHJd1ydvfsy7G3kFxodtL16Y VkA+I/vRDrfhluQ5utDoaoxN+uDpnyWlRZwO9KWNQfhPH7e1wXl2kfLdtIW8eyu0Ncy1 yKR56w2FVmB92qVhuDtkeVg1avcnaJIh4uHseI+LVhjVSen0auZGP7ZPMU3hNFnFkFeq 7600Sy+VcqjTifRFP1SoMXNcyZz2xYSA4cfbp72OCZtvE4hAy0e9JWyGYX20qCuQNVzi 6EZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-disposition:mime-version:references:subject:cc :cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:to:from:date:user-agent:message-id :dkim-signature; bh=Ivmf8KcU0nLZ7jKiUwDMxrV5SnY1/n4WH4SoklosWW0=; b=Uv4T0AYE91dfhCDv9iaBzaRzQ3teHdn90KHKQEtVNWFNbgfAN0I9QXz3QKsQJvbclI +ZTKSNJobF7q4Us7MNqrnJ7UoEYfGOq/cdO1dPBHFSk8lFYR31Z/k68sWGccGnyIPNCA OTItD7BodV/pZUoq4mg7b4GTga/07gFX3Jam2llA8zXQ4UC3f3sTdZmGZNbjQkVrKoZo 0F501eCX/cDbawjCJ7KzD5b9UyWOilyVW6wA9eRq7NiaOtzdMcJNRJX/rF41TKSlY7z5 uQOa6cOcTYogYqaohMvUhl92yYFDT/PT/3xarl04G1CnPPLnAV2YDQKvFp8dVf1943hu 67iw== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=OHCLLzJ6; spf=pass (google.com: domain of 01000167cd113a49-ada7b71e-0fa9-4897-b324-b7a2e1bd0f1c-000000@amazonses.com designates 54.240.9.35 as permitted sender) smtp.mailfrom=01000167cd113a49-ada7b71e-0fa9-4897-b324-b7a2e1bd0f1c-000000@amazonses.com Received: from a9-35.smtp-out.amazonses.com (a9-35.smtp-out.amazonses.com. [54.240.9.35]) by mx.google.com with ESMTPS id u13si2533036qvp.212.2018.12.20.11.21.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 20 Dec 2018 11:21:58 -0800 (PST) Received-SPF: pass (google.com: domain of 01000167cd113a49-ada7b71e-0fa9-4897-b324-b7a2e1bd0f1c-000000@amazonses.com designates 54.240.9.35 as permitted sender) client-ip=54.240.9.35; Authentication-Results: mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=OHCLLzJ6; spf=pass (google.com: domain of 01000167cd113a49-ada7b71e-0fa9-4897-b324-b7a2e1bd0f1c-000000@amazonses.com designates 54.240.9.35 as permitted sender) smtp.mailfrom=01000167cd113a49-ada7b71e-0fa9-4897-b324-b7a2e1bd0f1c-000000@amazonses.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1545333717; h=Message-Id:Date:From:To:Cc:Cc:Cc:CC:Cc:Cc:Cc:Cc:Cc:Cc:Cc:Subject:References:MIME-Version:Content-Type:Feedback-ID; bh=9RbM/jvYgrX3laKFEpjCMiN+ksJv3eWbjUkCaqrfwkE=; b=OHCLLzJ6oPFtcW68XCABgoUNOPVk6+3UQC2JsdbyVQyPQYljUakQzjMHr+tVlzug 2Vhpy7LZgA18mSKFfdg/iDG1ZWq+wKqvZZ43b68MiSQdWISkARuNc5UnrfGittJIxn7 ZDgfT1On7RLQBhryQyuP+CQHfiloPZZBUn9hI4OU= Message-ID: <01000167cd113a49-ada7b71e-0fa9-4897-b324-b7a2e1bd0f1c-000000@email.amazonses.com> User-Agent: quilt/0.65 Date: Thu, 20 Dec 2018 19:21:57 +0000 From: Christoph Lameter To: Matthew Wilcox Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: Pekka Enberg CC: akpm@linux-foundation.org Cc: Mel Gorman Cc: andi@firstfloor.org Cc: Rik van Riel Cc: Dave Chinner Cc: Christoph Hellwig Cc: Michal Hocko Cc: Mike Kravetz Subject: [RFC 2/7] slub: Add defrag_ratio field and sysfs support References: <20181220192145.023162076@linux.com> MIME-Version: 1.0 Content-Disposition: inline; filename=defrag_ratio X-SES-Outgoing: 2018.12.20-54.240.9.35 Feedback-ID: 1.us-east-1.fQZZZ0Xtj2+TD7V5apTT/NrT6QKuPgzCT/IC7XYgDKI=:AmazonSES X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP "defrag_ratio" is used to set the threshold at which defragmentation should be attempted on a slab page. "defrag_ratio" is percentage in the range of 1 - 100. If more than that percentage of slots in a slab page are unused the the slab page will become subject to defragmentation. Add a defrag ratio field and set it to 30% by default. A limit of 30% specifies that less than 3 out of 10 available slots for objects need to be leftover before slab defragmentation will be attempted on the remaining objects. Signed-off-by: Christoph Lameter --- Documentation/ABI/testing/sysfs-kernel-slab | 13 +++++++++++++ include/linux/slub_def.h | 6 ++++++ mm/slub.c | 23 +++++++++++++++++++++++ 3 files changed, 42 insertions(+) Index: linux/mm/slub.c =================================================================== --- linux.orig/mm/slub.c +++ linux/mm/slub.c @@ -3628,6 +3628,7 @@ static int kmem_cache_open(struct kmem_c set_cpu_partial(s); + s->defrag_ratio = 30; #ifdef CONFIG_NUMA s->remote_node_defrag_ratio = 1000; #endif @@ -5113,6 +5114,27 @@ static ssize_t destroy_by_rcu_show(struc } SLAB_ATTR_RO(destroy_by_rcu); +static ssize_t defrag_ratio_show(struct kmem_cache *s, char *buf) +{ + return sprintf(buf, "%d\n", s->defrag_ratio); +} + +static ssize_t defrag_ratio_store(struct kmem_cache *s, + const char *buf, size_t length) +{ + unsigned long ratio; + int err; + + err = kstrtoul(buf, 10, &ratio); + if (err) + return err; + + if (ratio < 100) + s->defrag_ratio = ratio; + return length; +} +SLAB_ATTR(defrag_ratio); + #ifdef CONFIG_SLUB_DEBUG static ssize_t slabs_show(struct kmem_cache *s, char *buf) { @@ -5437,6 +5459,7 @@ static struct attribute *slab_attrs[] = &validate_attr.attr, &alloc_calls_attr.attr, &free_calls_attr.attr, + &defrag_ratio_attr.attr, #endif #ifdef CONFIG_ZONE_DMA &cache_dma_attr.attr, Index: linux/Documentation/ABI/testing/sysfs-kernel-slab =================================================================== --- linux.orig/Documentation/ABI/testing/sysfs-kernel-slab +++ linux/Documentation/ABI/testing/sysfs-kernel-slab @@ -180,6 +180,19 @@ Description: list. It can be written to clear the current count. Available when CONFIG_SLUB_STATS is enabled. +What: /sys/kernel/slab/cache/defrag_ratio +Date: December 2018 +KernelVersion: 4.18 +Contact: Christoph Lameter + Pekka Enberg , +Description: + The defrag_ratio files allows the control of how agressive + slab fragmentation reduction works at reclaiming objects from + sparsely populated slabs. This is a percentage. If a slab + has more than this percentage of available object then reclaim + will attempt to reclaim objects so that the whole slab + page can be freed. The default is 30%. + What: /sys/kernel/slab/cache/deactivate_to_tail Date: February 2008 KernelVersion: 2.6.25 Index: linux/include/linux/slub_def.h =================================================================== --- linux.orig/include/linux/slub_def.h +++ linux/include/linux/slub_def.h @@ -104,6 +104,12 @@ struct kmem_cache { unsigned int red_left_pad; /* Left redzone padding size */ const char *name; /* Name (only for display!) */ struct list_head list; /* List of slab caches */ + int defrag_ratio; /* + * Ratio used to check the percentage of + * objects allocate in a slab page. + * If less than this ratio is allocated + * then reclaim attempts are made. + */ #ifdef CONFIG_SYSFS struct kobject kobj; /* For sysfs */ struct work_struct kobj_remove_work; From patchwork Thu Dec 20 19:21:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Christoph Lameter (Ampere)" X-Patchwork-Id: 10739431 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE7A214E5 for ; Thu, 20 Dec 2018 19:22:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAAE228CA1 for ; Thu, 20 Dec 2018 19:22:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9E74528D22; Thu, 20 Dec 2018 19:22:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2A0C928CA1 for ; Thu, 20 Dec 2018 19:22:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5165C8E000F; Thu, 20 Dec 2018 14:22:00 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4C3CE8E0001; Thu, 20 Dec 2018 14:22:00 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 319AE8E000F; Thu, 20 Dec 2018 14:22:00 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 05B6B8E0001 for ; Thu, 20 Dec 2018 14:22:00 -0500 (EST) Received: by mail-qt1-f198.google.com with SMTP id k90so2981600qte.0 for ; Thu, 20 Dec 2018 11:22:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:subject:references:mime-version :content-disposition:feedback-id; bh=ee1pZgnROCoNsOrAam7KrKG129xpjjIGO5hfhRbKwK0=; b=HVf6R2Fg8DzlD3BPkW4ktKgvFi3VCNG26uxLmuBXIKFyA2waZ22ffOiV4Y1E/CzaTo UBOed6/Mgi+N/cL9R3FzC2fRxLTDsibv8iOdlb3skwd/8vcCbERrfKkgzq3dMfzq8iby qp0SB6lJnTngmA5ZDATXXSOGZy3cFKwZoqGYcG1dwMwVO8CktyZzmd+jdbmJgSbYMADc 4XJ4kvUBDL3bMYM47v3PGthJfXLnQq7yzFlhl4rIJaQRTBpMcI8xv57BBYXWRlNz8qpE 8PLP1H3FRhfWxPNBzrfE5TsoDVH8FbBN5aOya8h5QIU2Wqx7gHLbZPSJ+YN+6vFj2nTc 0geA== X-Gm-Message-State: AA+aEWaR1+gd3L831oBpN07VEF9ZYSIPuiJ81/2NXbuKc1+9J7wQGLnH 3ayx6j/TXap7SMfg0GvspEzCueJATPiOLclIHAKLflzVKKh+6/SbHKSSm6+2JmZibcnPtFdqhfl BzxfNKaEPuxIPx2Uz2A/mFZSyGwSFoqhKMOFwFSgpiGfZA9blar5wyYfjnAHOOS8= X-Received: by 2002:a0c:e394:: with SMTP id a20mr26654212qvl.42.1545333719812; Thu, 20 Dec 2018 11:21:59 -0800 (PST) X-Google-Smtp-Source: AFSGD/XZch4Hcz5r0QesBu8kkWEy4r2vWvbJ1E/Ygl/btKQLc6lNbN5C+PA4iMAQIv6okCZCOo8D X-Received: by 2002:a0c:e394:: with SMTP id a20mr26654181qvl.42.1545333719396; Thu, 20 Dec 2018 11:21:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545333719; cv=none; d=google.com; s=arc-20160816; b=lbyzUAiffQec4a2d8tV9OEu/oMH2Fip9qYj7eausO8tQqk452Jq+2sI4M3ExsEM1HG LpviYS+VKIHQqRuWYE5E+2Sbibq22xqiu8RVet1cJjn3ft6u5NZ+1e8dIgbOHi8fZchZ npqZkyeuXHTLMOsD9ExvpMBj86YkotYrKzVfDNcD5bsnzE8OQ1MmhBg9/nu3PiYic5an yNiJsW2CUGgmUXDp0giHiYVO8WoPhzamjrLsX6CKM772s5UT5A5tjcEegAhFI+qeR0uG q3gXyngbOXxu+Sne5BCcCYxIR1iXRcYc5CNgS7wZ8B5cJKun5H9Xbzh2AJq6QWa/wmrp pkSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-disposition:mime-version:references:subject:cc :cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:to:from:date:user-agent:message-id :dkim-signature; bh=ee1pZgnROCoNsOrAam7KrKG129xpjjIGO5hfhRbKwK0=; b=kiTemvmzGHiYeN5QJb2bdB27ORDQZCktANkeVE8DgKl9MKKqHyPhPr871TON1gyRzy nEM6PeVVsZmOTTAnT5gj0fMa3xSomCXFuZy8htAXMhyYUm4qBjykC/xul5N/MTi0Jwp2 eTYyyafTi06wcWIvQlRV9xHPP5nm+0lFvMFK+7CplxqPNV/Ud/kZtxnugRsCVKSZz/aQ Sxx/KvLcn6yQ2lbAn3oHUsI/Xd9nbURqD1LPMD3/b7EkY4noNhMK/AZzMphiS87cGzRr rT3QilUe4fKRqUTQ2f83a5+bY8CSkqSmKWM0XcqTJbHByeH6HGgCwPdrioAiW53yTA3O SN8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=QlNAG2Db; spf=pass (google.com: domain of 01000167cd113f16-b5a2b2db-e75a-4bad-a47a-0a66fbf7fd8a-000000@amazonses.com designates 54.240.9.32 as permitted sender) smtp.mailfrom=01000167cd113f16-b5a2b2db-e75a-4bad-a47a-0a66fbf7fd8a-000000@amazonses.com Received: from a9-32.smtp-out.amazonses.com (a9-32.smtp-out.amazonses.com. [54.240.9.32]) by mx.google.com with ESMTPS id t7si3981848qvh.32.2018.12.20.11.21.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 20 Dec 2018 11:21:59 -0800 (PST) Received-SPF: pass (google.com: domain of 01000167cd113f16-b5a2b2db-e75a-4bad-a47a-0a66fbf7fd8a-000000@amazonses.com designates 54.240.9.32 as permitted sender) client-ip=54.240.9.32; Authentication-Results: mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=QlNAG2Db; spf=pass (google.com: domain of 01000167cd113f16-b5a2b2db-e75a-4bad-a47a-0a66fbf7fd8a-000000@amazonses.com designates 54.240.9.32 as permitted sender) smtp.mailfrom=01000167cd113f16-b5a2b2db-e75a-4bad-a47a-0a66fbf7fd8a-000000@amazonses.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1545333718; h=Message-Id:Date:From:To:Cc:Cc:Cc:CC:Cc:Cc:Cc:Cc:Cc:Cc:Cc:Subject:References:MIME-Version:Content-Type:Feedback-ID; bh=MEo1JnulmTAHG+WXA+afG0E08WmvUdsEOe37fi6PrrM=; b=QlNAG2DbLCddJR2rTzuDDeU5Bi4NhHZ686TqNEwNlRTD/c2MgN5RbdIK98ycS8Cu 8Cb2xj8r43L7Rx9lSoR4+7luOLmsh4gqghmW/FkPslBSSU9RuEk5Zi45gxjTUtrsfqu 5QH9n+9YHIP7rGhBd5cPpCWa/SItiQKh8huHJC7g= Message-ID: <01000167cd113f16-b5a2b2db-e75a-4bad-a47a-0a66fbf7fd8a-000000@email.amazonses.com> User-Agent: quilt/0.65 Date: Thu, 20 Dec 2018 19:21:58 +0000 From: Christoph Lameter To: Matthew Wilcox Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: Pekka Enberg CC: akpm@linux-foundation.org Cc: Mel Gorman Cc: andi@firstfloor.org Cc: Rik van Riel Cc: Dave Chinner Cc: Christoph Hellwig Cc: Michal Hocko Cc: Mike Kravetz Subject: [RFC 3/7] slub: Add isolate() and migrate() methods References: <20181220192145.023162076@linux.com> MIME-Version: 1.0 Content-Disposition: inline; filename=isolate_and_migrate_methods X-SES-Outgoing: 2018.12.20-54.240.9.32 Feedback-ID: 1.us-east-1.fQZZZ0Xtj2+TD7V5apTT/NrT6QKuPgzCT/IC7XYgDKI=:AmazonSES X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Add the two methods needed for moving objects and enable the display of the callbacks via the /sys/kernel/slab interface. Add documentation explaining the use of these methods and the prototypes for slab.h. Add functions to setup the callbacks method for a slab cache. Add empty functions for SLAB/SLOB. The API is generic so it could be theoretically implemented for these allocators as well. Signed-off-by: Christoph Lameter --- include/linux/slab.h | 50 +++++++++++++++++++++++++++++++++++++++++++++++ include/linux/slub_def.h | 3 ++ mm/slub.c | 29 ++++++++++++++++++++++++++- 3 files changed, 81 insertions(+), 1 deletion(-) Index: linux/include/linux/slub_def.h =================================================================== --- linux.orig/include/linux/slub_def.h +++ linux/include/linux/slub_def.h @@ -99,6 +99,9 @@ struct kmem_cache { gfp_t allocflags; /* gfp flags to use on each alloc */ int refcount; /* Refcount for slab cache destroy */ void (*ctor)(void *); + kmem_isolate_func *isolate; + kmem_migrate_func *migrate; + unsigned int inuse; /* Offset to metadata */ unsigned int align; /* Alignment */ unsigned int red_left_pad; /* Left redzone padding size */ Index: linux/mm/slub.c =================================================================== --- linux.orig/mm/slub.c +++ linux/mm/slub.c @@ -3498,7 +3498,6 @@ static int calculate_sizes(struct kmem_c else s->flags &= ~__OBJECT_POISON; - /* * If we are Redzoning then check if there is some space between the * end of the object and the free pointer. If not then add an @@ -4311,6 +4310,25 @@ int __kmem_cache_create(struct kmem_cach return err; } +void kmem_cache_setup_mobility(struct kmem_cache *s, + kmem_isolate_func isolate, kmem_migrate_func migrate) +{ + /* + * Defragmentable slabs must have a ctor otherwise objects may be + * in an undetermined state after they are allocated. + */ + BUG_ON(!s->ctor); + s->isolate = isolate; + s->migrate = migrate; + /* + * Sadly serialization requirements currently mean that we have + * to disable fast cmpxchg based processing. + */ + s->flags &= ~__CMPXCHG_DOUBLE; + +} +EXPORT_SYMBOL(kmem_cache_setup_mobility); + void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller) { struct kmem_cache *s; @@ -5004,6 +5022,20 @@ static ssize_t ops_show(struct kmem_cach if (s->ctor) x += sprintf(buf + x, "ctor : %pS\n", s->ctor); + + if (s->isolate) { + x += sprintf(buf + x, "isolate : "); + x += sprint_symbol(buf + x, + (unsigned long)s->isolate); + x += sprintf(buf + x, "\n"); + } + + if (s->migrate) { + x += sprintf(buf + x, "migrate : "); + x += sprint_symbol(buf + x, + (unsigned long)s->migrate); + x += sprintf(buf + x, "\n"); + } return x; } SLAB_ATTR_RO(ops); Index: linux/include/linux/slab.h =================================================================== --- linux.orig/include/linux/slab.h +++ linux/include/linux/slab.h @@ -153,6 +153,68 @@ void memcg_deactivate_kmem_caches(struct void memcg_destroy_kmem_caches(struct mem_cgroup *); /* + * Function prototypes passed to kmem_cache_setup_mobility() to enable mobile + * objects and targeted reclaim in slab caches. + */ + +/* + * kmem_cache_isolate_func() is called with locks held so that the slab + * objects cannot be freed. We are in an atomic context and no slab + * operations may be performed. The purpose of kmem_cache_isolate_func() + * is to pin the object so that it cannot be freed until + * kmem_cache_migrate_func() has processed them. This may be accomplished + * by increasing the refcount or setting a flag. + * + * Parameters passed are the number of objects to process and an array of + * pointers to objects which are intended to be moved. + * + * Returns a pointer that is passed to the migrate function. If any objects + * cannot be touched at this point then the pointer may indicate a + * failure and then the migration function can simply remove the references + * that were already obtained. The private data could be used to track + * the objects that were already pinned. + * + * The object pointer array passed is also passed to kmem_cache_migrate(). + * The function may remove objects from the array by setting pointers to + * NULL. This is useful if we can determine that an object is being freed + * because kmem_cache_isolate_func() was called when the subsystem + * was calling kmem_cache_free(). + * In that case it is not necessary to increase the refcount or + * specially mark the object because the release of the slab lock + * will lead to the immediate freeing of the object. + */ +typedef void *kmem_isolate_func(struct kmem_cache *, void **, int); + +/* + * kmem_cache_move_migrate_func is called with no locks held and interrupts + * enabled. Sleeping is possible. Any operation may be performed in + * migrate(). kmem_cache_migrate_func should allocate new objects and + * free all the objects. + ** + * Parameters passed are the number of objects in the array, the array of + * pointers to the objects, the NUMA node where the object should be + * allocated and the pointer returned by kmem_cache_isolate_func(). + * + * Success is checked by examining the number of remaining objects in + * the slab. If the number is zero then the objects will be freed. + */ +typedef void kmem_migrate_func(struct kmem_cache *, void **, int nr, int node, void *private); + +/* + * kmem_cache_setup_mobility() is used to setup callbacks for a slab cache. + */ +#ifdef CONFIG_SLUB +void kmem_cache_setup_mobility(struct kmem_cache *, kmem_isolate_func, + kmem_migrate_func); +#else +static inline void kmem_cache_setup_mobility(struct kmem_cache *s, + kmem_isolate_func isolate, kmem_migrate_func migrate) {} +#endif + +/* + * Allocator specific definitions. These are mainly used to establish optimized + * ways to convert kmalloc() calls to kmem_cache_alloc() invocations by + * selecting the appropriate general cache at compile time. * Please use this macro to create slab caches. Simply specify the * name of the structure and maybe some flags that are listed above. * Index: linux/mm/slab_common.c =================================================================== --- linux.orig/mm/slab_common.c +++ linux/mm/slab_common.c @@ -298,7 +298,7 @@ int slab_unmergeable(struct kmem_cache * if (!is_root_cache(s)) return 1; - if (s->ctor) + if (s->ctor || s->isolate || s->migrate) return 1; if (s->usersize) From patchwork Thu Dec 20 19:22:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Christoph Lameter (Ampere)" X-Patchwork-Id: 10739433 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 526D4746 for ; Thu, 20 Dec 2018 19:22:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4075128CA1 for ; Thu, 20 Dec 2018 19:22:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3409D28D22; Thu, 20 Dec 2018 19:22:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D323528CA1 for ; Thu, 20 Dec 2018 19:22:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6C1828E0010; Thu, 20 Dec 2018 14:22:01 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 69B0D8E0001; Thu, 20 Dec 2018 14:22:01 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 539988E0010; Thu, 20 Dec 2018 14:22:01 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 265B38E0001 for ; Thu, 20 Dec 2018 14:22:01 -0500 (EST) Received: by mail-qt1-f199.google.com with SMTP id n50so2961039qtb.9 for ; Thu, 20 Dec 2018 11:22:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:subject:references:mime-version :content-disposition:feedback-id; bh=tMJflD5NVJ4V9q8P29WkibQrcq00hjoERQHQ4TCQo7A=; b=q+U9e30pMG2MWjz29xtMQQHrpcR71IuRyWpFf0Ex1ctdE0e5NIn8TJqLqr8KyLa3Us q/1W7I++xepZzIJlfMV79vMoW1S0JIKBf0ewc0gqveAD6K8lBBDwT+IaBzxS00imlJz4 tkCAwiYrqMdSIFy6V7FuVi9F57QYHFn2Gb1e1iyusA7lgf6xQtnkhngspEp/xiSoWgr4 IHjP74yZ2ygDQyGN9YmAsL3k3MUtpqiEP5rw1wvTF36mGAwx4r5PsDo/u+GgKHW8WFsP hZWp2v2yVBvmAPYq4uq09cwsL5OCJ28IwAToBDWrWrBrLsBa8tTCzBtx8C2gQ/s2iOZq 9M7Q== X-Gm-Message-State: AA+aEWbh1AeBNglMwDuTL/Gj8F5qo0ZvUCAT7nqQv8MIuCEM83bDXTHv UoC9FukdOxf5FGHrJZmfGA82ImZwVooBzLOnDyyiNgeYWcf6PkkRk1kj0aRMGtPsZ06Jh5EZO0P 31FTira2T9yWkHM77tqdRCpMyVmYHpe8h9U9h86JAXA3yjYx7oxd/pGBrKQKxHa0= X-Received: by 2002:a37:8383:: with SMTP id f125mr25961902qkd.49.1545333720943; Thu, 20 Dec 2018 11:22:00 -0800 (PST) X-Google-Smtp-Source: AFSGD/WjydxmlaK8juwpKqFmnUwLM4TEysp5Er1Uuu4D26HwP1TawBQdZoSZ9XNIsOCe+HNXqVEk X-Received: by 2002:a37:8383:: with SMTP id f125mr25961877qkd.49.1545333720455; Thu, 20 Dec 2018 11:22:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545333720; cv=none; d=google.com; s=arc-20160816; b=nTfW7x5wPrBU86NGEoF3uvrGyxRMu66vNjZ1fzO3+OeQYxkHWASUkMY0sTzf1zLByl DwAEh4Xl4oJUVa6EoCrFGJjNvCniEa763qwunJRzU/7IDzJrAIJqidd8pCv16uMZWc2b HsprzvL0zQdhF8A4aLAXmebOVqtQTXTtsUEtbvtw8+Mkl50WxPSiKjKVNwkmnQrXDwB1 z6I9QogkeetfIH/8tTegJOHsfkfzBtIEa9+nlhTA5f0kUU6zklcu2reehX9YKo7GUmWH xCrn78JmdmzvTX4h1uO/MkXelN8+CIKH8n7riIkO3d/kgFVownD72RSpwBIROVrU9dY8 +Aig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-disposition:mime-version:references:subject:cc :cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:to:from:date:user-agent:message-id :dkim-signature; bh=tMJflD5NVJ4V9q8P29WkibQrcq00hjoERQHQ4TCQo7A=; b=TwohWNKO1xj89mTonfNJ0IAOiaiy0vGZeH/wlQVV3g0GhLvYl9sBx3wsNQoi3XHkyh 7igrhrmL01ipFCDPn3lpa9aK+8v9GwUie8aGY2Gjv8OIMVHtasaWx9f5VkvOiNbfbgmH Rj05ZXGGzS6w8ZX4yjUFd0BL0mLIfKSe1rrxSydnwmNRyca8yflQptE5Fu4YenBXq/TJ 2+LDTlV+ZbwIHjilpI7qiZAB/NC4onHnuPT8fWc+PAH1nQXmiC1KI5izNCaVwApbmb9G kWPYLU45qXCnQ+DlxtSfHNBYHzHgD0kIBxIjGT5FKUHHTm14S/RWQs8GUMn/UH8F04hO 5S9w== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=PfolZphx; spf=pass (google.com: domain of 01000167cd1143e3-1533fccc-7036-4a4e-97ea-5be8b347bbf0-000000@amazonses.com designates 54.240.9.34 as permitted sender) smtp.mailfrom=01000167cd1143e3-1533fccc-7036-4a4e-97ea-5be8b347bbf0-000000@amazonses.com Received: from a9-34.smtp-out.amazonses.com (a9-34.smtp-out.amazonses.com. [54.240.9.34]) by mx.google.com with ESMTPS id s188si603132qkh.260.2018.12.20.11.22.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 20 Dec 2018 11:22:00 -0800 (PST) Received-SPF: pass (google.com: domain of 01000167cd1143e3-1533fccc-7036-4a4e-97ea-5be8b347bbf0-000000@amazonses.com designates 54.240.9.34 as permitted sender) client-ip=54.240.9.34; Authentication-Results: mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=PfolZphx; spf=pass (google.com: domain of 01000167cd1143e3-1533fccc-7036-4a4e-97ea-5be8b347bbf0-000000@amazonses.com designates 54.240.9.34 as permitted sender) smtp.mailfrom=01000167cd1143e3-1533fccc-7036-4a4e-97ea-5be8b347bbf0-000000@amazonses.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1545333720; h=Message-Id:Date:From:To:Cc:Cc:Cc:CC:Cc:Cc:Cc:Cc:Cc:Cc:Cc:Subject:References:MIME-Version:Content-Type:Feedback-ID; bh=HPynk5ie8DqlrwTSrcpoX8ICqEk/Juei7kWbVieJc3U=; b=PfolZphxIyAKtwwsB+etH1iPIkc5vgB18IIUZeZFp+eNrdEkSPSxmjzp2/HrWYBD 6jNi+e3ZPWy1bh34NwFCLIQXbD4VJEqXD6sowCDjyMi/oR/JGbPKLJ78ZjXGhVt4Prf +vNus1fbj5+deIsZuLLB5qaBnqanDwkpGAKDAHOo= Message-ID: <01000167cd1143e3-1533fccc-7036-4a4e-97ea-5be8b347bbf0-000000@email.amazonses.com> User-Agent: quilt/0.65 Date: Thu, 20 Dec 2018 19:22:00 +0000 From: Christoph Lameter To: Matthew Wilcox Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: Pekka Enberg CC: akpm@linux-foundation.org Cc: Mel Gorman Cc: andi@firstfloor.org Cc: Rik van Riel Cc: Dave Chinner Cc: Christoph Hellwig Cc: Michal Hocko Cc: Mike Kravetz Subject: [RFC 4/7] slub: Sort slab cache list and establish maximum objects for defrag slabs References: <20181220192145.023162076@linux.com> MIME-Version: 1.0 Content-Disposition: inline; filename=sort_and_max X-SES-Outgoing: 2018.12.20-54.240.9.34 Feedback-ID: 1.us-east-1.fQZZZ0Xtj2+TD7V5apTT/NrT6QKuPgzCT/IC7XYgDKI=:AmazonSES X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It is advantageous to have all defragmentable slabs together at the beginning of the list of slabs so that there is no need to scan the complete list. Put defragmentable caches first when adding a slab cache and others last. Determine the maximum number of objects in defragmentable slabs. This allows the sizing of the array holding refs to objects in a slab later. Signed-off-by: Christoph Lameter --- mm/slub.c | 26 ++++++++++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) Index: linux/mm/slub.c =================================================================== --- linux.orig/mm/slub.c +++ linux/mm/slub.c @@ -196,6 +196,9 @@ static inline bool kmem_cache_has_cpu_pa /* Use cmpxchg_double */ #define __CMPXCHG_DOUBLE ((slab_flags_t __force)0x40000000U) +/* Maximum objects in defragmentable slabs */ +static unsigned int max_defrag_slab_objects; + /* * Tracking user of a slab. */ @@ -4310,22 +4313,45 @@ int __kmem_cache_create(struct kmem_cach return err; } +/* + * Allocate a slab scratch space that is sufficient to keep at least + * max_defrag_slab_objects pointers to individual objects and also a bitmap + * for max_defrag_slab_objects. + */ +static inline void *alloc_scratch(void) +{ + return kmalloc(max_defrag_slab_objects * sizeof(void *) + + BITS_TO_LONGS(max_defrag_slab_objects) * sizeof(unsigned long), + GFP_KERNEL); +} + void kmem_cache_setup_mobility(struct kmem_cache *s, kmem_isolate_func isolate, kmem_migrate_func migrate) { + int max_objects = oo_objects(s->max); + /* * Defragmentable slabs must have a ctor otherwise objects may be * in an undetermined state after they are allocated. */ BUG_ON(!s->ctor); + + mutex_lock(&slab_mutex); + s->isolate = isolate; s->migrate = migrate; + /* * Sadly serialization requirements currently mean that we have * to disable fast cmpxchg based processing. */ s->flags &= ~__CMPXCHG_DOUBLE; + list_move(&s->list, &slab_caches); /* Move to top */ + if (max_objects > max_defrag_slab_objects) + max_defrag_slab_objects = max_objects; + + mutex_unlock(&slab_mutex); } EXPORT_SYMBOL(kmem_cache_setup_mobility); Index: linux/mm/slab_common.c =================================================================== --- linux.orig/mm/slab_common.c +++ linux/mm/slab_common.c @@ -393,7 +393,7 @@ static struct kmem_cache *create_cache(c goto out_free_cache; s->refcount = 1; - list_add(&s->list, &slab_caches); + list_add_tail(&s->list, &slab_caches); memcg_link_cache(s); out: if (err) From patchwork Thu Dec 20 19:22:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Christoph Lameter (Ampere)" X-Patchwork-Id: 10739435 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A90E0746 for ; Thu, 20 Dec 2018 19:22:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9431F28CA1 for ; Thu, 20 Dec 2018 19:22:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8841E28D22; Thu, 20 Dec 2018 19:22:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CD04428CA1 for ; Thu, 20 Dec 2018 19:22:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1EC98E0012; Thu, 20 Dec 2018 14:22:02 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 983A98E0001; Thu, 20 Dec 2018 14:22:02 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7D76A8E0012; Thu, 20 Dec 2018 14:22:02 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by kanga.kvack.org (Postfix) with ESMTP id 4613D8E0001 for ; Thu, 20 Dec 2018 14:22:02 -0500 (EST) Received: by mail-qt1-f199.google.com with SMTP id m37so2944163qte.10 for ; Thu, 20 Dec 2018 11:22:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:subject:references:mime-version :content-disposition:feedback-id; bh=9xsw+LRXhN6pULsUawdWy8PNdT5DJqPqassI+W8RJZA=; b=W3gNfbEWLB3Kyr+qQdKJwUjWPjPbROFBzHEjnj8Pnbw1+aHUPetRrfscIesaOMRyuj Sk9lAQ36gklOXf9bcTpQYkBxzvrMV0G78ylYQ30yn2/IJsfSlLWNkyZc5GZb3AgmxV/v LlWItfDtARmuex6bE6hf8qy2sBKqxIL6xuep9xDNWIBMgPYKdSQEIEb7/OfaxXnoaYPP 2NwxVJPnKp5eH68ZSrV8/YBSg8XMhWacR64useOqwa++fiCQsE899KG5oN7QHyTs5Y5d NBZCpgZLZra6sHIlAndqI8tw5O28b0G5h/iVK1mw0fSAhQGaZPQedN2LP9peRwqE3BuY 5ohw== X-Gm-Message-State: AA+aEWYYO760JF9xYx7vjp0amYNwQVdu916DDSNNLGZAXDzypS2CfRnT /vx+JxKltMm9PBpNPG/2hQ6t90b+1B2CF7pcrN/neSmMWtSDejTqVSghYZwBLOuqyMR/994Fv6w 1LM7AVqBpvt9BElnpJUymLYic14cRRZ4iMi7qgIs3/7D6NivAdhLNBF+2YnFDgjU= X-Received: by 2002:aed:3c7d:: with SMTP id u58mr28254982qte.198.1545333722046; Thu, 20 Dec 2018 11:22:02 -0800 (PST) X-Google-Smtp-Source: AFSGD/UXQWr/SSbuabKSfCzlcKisKVo7nl3+3Wml7LQNT73o1FzIAX2W6vE+iQpXM/pjLPZeER/6 X-Received: by 2002:aed:3c7d:: with SMTP id u58mr28254958qte.198.1545333721636; Thu, 20 Dec 2018 11:22:01 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545333721; cv=none; d=google.com; s=arc-20160816; b=0yQRsXn3SNhyxrEv5Bvh5upj22yFcrBNF9BH3xE5I6GecwX735BSJrPFmw+HMcdZJK 0K5QkOzaI6ABVcaNvjCHmw2mYQjmVqZDqxWdkBToLsaaIqOeqsdKLVDKAP5qPsSZ5SqE iwiSfJ2TYDLJhghwlbvqAozcFQNKQygTO76bOL2vidvaueSy/1KEnS68ZDmBFProKbVc 8ce8rG5ktLkur9P50Qs3ba7+GUDOAO4fTzHIt9aSFudIoI3CYcLvIztzWUczXGEII1LK nDI2viDDk6ZhUECj2MDYTTDgOjgpMe0Hr+ao6wpwXSL6dMR4qXG5jifagaN+DQI9mSOk uX4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-disposition:mime-version:references:subject:cc :cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:to:from:date:user-agent:message-id :dkim-signature; bh=9xsw+LRXhN6pULsUawdWy8PNdT5DJqPqassI+W8RJZA=; b=prFlyGG9HZVdIBYHc6TvsSFM6l2W0GAwps15PZAWGw8IXM29ATTv3dFJyT+TuJPNHU h9pOCiT1G8nTJ+4SIRUofCQAIHl1wduWOmqy/hEI/2gnqRFB1B2V0B0KTQADvlefb+A4 hZ5LQlbjtRKTqLkd5aW3XDb1JgLIQ9CJ6Y7kdN77gcIYjwkwpQ76zYD68guOr1EK2k5j MNohMv60x1V4TRMOGQtvMKabJ8NdA2zwtF5lXsZmHR3QrTB5RPWkEu1RYNctdfTsEwIw rFMb4MaxtNks6KNDfyMIeo6Z/cHPovpwFZypg8FpB9gb8jF3z3bN6uWzoe/+CueNqgQH 3h1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=XFRpu48z; spf=pass (google.com: domain of 01000167cd1147f1-b4c1de14-b084-405f-a4d4-26e0506d342e-000000@amazonses.com designates 54.240.9.34 as permitted sender) smtp.mailfrom=01000167cd1147f1-b4c1de14-b084-405f-a4d4-26e0506d342e-000000@amazonses.com Received: from a9-34.smtp-out.amazonses.com (a9-34.smtp-out.amazonses.com. [54.240.9.34]) by mx.google.com with ESMTPS id o24si4207390qkl.147.2018.12.20.11.22.01 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 20 Dec 2018 11:22:01 -0800 (PST) Received-SPF: pass (google.com: domain of 01000167cd1147f1-b4c1de14-b084-405f-a4d4-26e0506d342e-000000@amazonses.com designates 54.240.9.34 as permitted sender) client-ip=54.240.9.34; Authentication-Results: mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=XFRpu48z; spf=pass (google.com: domain of 01000167cd1147f1-b4c1de14-b084-405f-a4d4-26e0506d342e-000000@amazonses.com designates 54.240.9.34 as permitted sender) smtp.mailfrom=01000167cd1147f1-b4c1de14-b084-405f-a4d4-26e0506d342e-000000@amazonses.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1545333721; h=Message-Id:Date:From:To:Cc:Cc:Cc:CC:Cc:Cc:Cc:Cc:Cc:Cc:Cc:Subject:References:MIME-Version:Content-Type:Feedback-ID; bh=/ZPhvHYHxnhQXiDFBeTlfZ9rpP4ThfXAI7UKJ2h1uV4=; b=XFRpu48zIQe/ov7J8L8ah9XuPoxvby9CaIwWUnunOEf3Hz20fCT1tgxRWbjhwVOn 7SzDaFQLjcG+7EW7Zu1gfetTALDAOjUJR4TxHhhhg5wcg3crLpfCgw9HxpwTlw/IAbV XVFuD6zVkyWlL1HbGaIa2R1UgQVhyGAQXWLT0clM= Message-ID: <01000167cd1147f1-b4c1de14-b084-405f-a4d4-26e0506d342e-000000@email.amazonses.com> User-Agent: quilt/0.65 Date: Thu, 20 Dec 2018 19:22:01 +0000 From: Christoph Lameter To: Matthew Wilcox Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: Pekka Enberg CC: akpm@linux-foundation.org Cc: Mel Gorman Cc: andi@firstfloor.org Cc: Rik van Riel Cc: Dave Chinner Cc: Christoph Hellwig Cc: Michal Hocko Cc: Mike Kravetz Subject: [RFC 5/7] slub: Slab defrag core References: <20181220192145.023162076@linux.com> MIME-Version: 1.0 Content-Disposition: inline; filename=mobility_core X-SES-Outgoing: 2018.12.20-54.240.9.34 Feedback-ID: 1.us-east-1.fQZZZ0Xtj2+TD7V5apTT/NrT6QKuPgzCT/IC7XYgDKI=:AmazonSES X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Slab defragmentation may occur: 1. Unconditionally when kmem_cache_shrink is called on a slab cache by the kernel calling kmem_cache_shrink. 2. Through the use of the slabinfo command. 3. Per node defrag conditionally when kmem_cache_defrag() is called (can be called from reclaim code with a later patch). Defragmentation is only performed if the fragmentation of the slab is lower than the specified percentage. Fragmentation ratios are measured by calculating the percentage of objects in use compared to the total number of objects that the slab page can accomodate. The scanning of slab caches is optimized because the defragmentable slabs come first on the list. Thus we can terminate scans on the first slab encountered that does not support defragmentation. kmem_cache_defrag() takes a node parameter. This can either be -1 if defragmentation should be performed on all nodes, or a node number. A couple of functions must be setup via a call to kmem_cache_setup_defrag() in order for a slabcache to support defragmentation. These are kmem_defrag_isolate_func (void *isolate(struct kmem_cache *s, void **objects, int nr)) Must stabilize that the objects and ensure that they will not be freed until the migration function is complete. SLUB guarantees that the objects are still allocated. However, other threads may be blocked in slab_free() attempting to free objects in the slab. These may succeed as soon as isolate() returns to the slab allocator. The function must be able to detect such situations and void the attempts to free such objects (by for example voiding the corresponding entry in the objects array). No slab operations may be performed in isolate(). Interrupts are disabled. What can be done is very limited. The slab lock for the page that contains the object is taken. Any attempt to perform a slab operation may lead to a deadlock. kmem_defrag_isolate_func returns a private pointer that is passed to kmem_defrag_kick_func(). Should we be unable to obtain all references then that pointer may indicate to the kick() function that it should not attempt any object removal or move but simply undo the measure that were used to stabilize the object. kmem_defrag_migrate_func (void migrate(struct kmem_cache *, void **objects, int nr, int node, void *get_result)) After SLUB has stabilzed the objects in a slab it will then drop all locks and use migrate() to move objects out of the slab. The existence of the object is guaranteed by virtue of the earlier obtained references via kmem_defrag_get_func(). The callback may perform any slab operation since no locks are held at the time of call. The callback should remove the object from the slab in some way. This may be accomplished by reclaiming the object and then running kmem_cache_free() or reallocating it and then running kmem_cache_free(). Reallocation is advantageous because the partial slabs were just sorted to have the partial slabs with the most objects first. Reallocation is likely to result in filling up a slab in addition to freeing up one slab. A filled up slab can also be removed from the partial list. So there could be a double effect. kmem_defrag_migrate_func() does not return a result. SLUB will check the number of remaining objects in the slab. If all objects were removed then the slab is freed and we have reduced the overall fragmentation of the slab cache. Signed-off-by: Christoph Lameter --- include/linux/slab.h | 3 mm/slub.c | 265 ++++++++++++++++++++++++++++++++++++++++----------- 2 files changed, 215 insertions(+), 53 deletions(-) Index: linux/mm/slub.c =================================================================== --- linux.orig/mm/slub.c +++ linux/mm/slub.c @@ -351,6 +351,12 @@ static __always_inline void slab_lock(st bit_spin_lock(PG_locked, &page->flags); } +static __always_inline int slab_trylock(struct page *page) +{ + VM_BUG_ON_PAGE(PageTail(page), page); + return bit_spin_trylock(PG_locked, &page->flags); +} + static __always_inline void slab_unlock(struct page *page) { VM_BUG_ON_PAGE(PageTail(page), page); @@ -3946,79 +3952,6 @@ void kfree(const void *x) } EXPORT_SYMBOL(kfree); -#define SHRINK_PROMOTE_MAX 32 - -/* - * kmem_cache_shrink discards empty slabs and promotes the slabs filled - * up most to the head of the partial lists. New allocations will then - * fill those up and thus they can be removed from the partial lists. - * - * The slabs with the least items are placed last. This results in them - * being allocated from last increasing the chance that the last objects - * are freed in them. - */ -int __kmem_cache_shrink(struct kmem_cache *s) -{ - int node; - int i; - struct kmem_cache_node *n; - struct page *page; - struct page *t; - struct list_head discard; - struct list_head promote[SHRINK_PROMOTE_MAX]; - unsigned long flags; - int ret = 0; - - flush_all(s); - for_each_kmem_cache_node(s, node, n) { - INIT_LIST_HEAD(&discard); - for (i = 0; i < SHRINK_PROMOTE_MAX; i++) - INIT_LIST_HEAD(promote + i); - - spin_lock_irqsave(&n->list_lock, flags); - - /* - * Build lists of slabs to discard or promote. - * - * Note that concurrent frees may occur while we hold the - * list_lock. page->inuse here is the upper limit. - */ - list_for_each_entry_safe(page, t, &n->partial, lru) { - int free = page->objects - page->inuse; - - /* Do not reread page->inuse */ - barrier(); - - /* We do not keep full slabs on the list */ - BUG_ON(free <= 0); - - if (free == page->objects) { - list_move(&page->lru, &discard); - n->nr_partial--; - } else if (free <= SHRINK_PROMOTE_MAX) - list_move(&page->lru, promote + free - 1); - } - - /* - * Promote the slabs filled up most to the head of the - * partial list. - */ - for (i = SHRINK_PROMOTE_MAX - 1; i >= 0; i--) - list_splice(promote + i, &n->partial); - - spin_unlock_irqrestore(&n->list_lock, flags); - - /* Release empty slabs */ - list_for_each_entry_safe(page, t, &discard, lru) - discard_slab(s, page); - - if (slabs_node(s, node)) - ret = 1; - } - - return ret; -} - #ifdef CONFIG_MEMCG static void kmemcg_cache_deact_after_rcu(struct kmem_cache *s) { @@ -4325,14 +4258,271 @@ static inline void *alloc_scratch(void) GFP_KERNEL); } +/* + * Move all objects in the given slab. + * + * If the target node is the current node then the object is moved else + * where on the same node. Which is an effective way of defragmentation + * since the current slab page with its object is exempt from allocation. + * + * The scratch area passed to list function is sufficient to hold + * struct listhead times objects per slab. We use it to hold void ** times + * objects per slab plus a bitmap for each object. + */ +static void kmem_cache_move(struct page *page, void *scratch, int node) +{ + void **vector = scratch; + void *p; + void *addr = page_address(page); + struct kmem_cache *s; + unsigned long *map; + int count; + void *private; + unsigned long flags; + unsigned long objects; + + local_irq_save(flags); + slab_lock(page); + + BUG_ON(!PageSlab(page)); /* Must be s slab page */ + BUG_ON(!page->frozen); /* Slab must have been frozen earlier */ + + s = page->slab_cache; + objects = page->objects; + map = scratch + objects * sizeof(void **); + + /* Determine used objects */ + bitmap_fill(map, objects); + for (p = page->freelist; p; p = get_freepointer(s, p)) + __clear_bit(slab_index(p, s, addr), map); + + /* Build vector of pointers to objects */ + count = 0; + memset(vector, 0, objects * sizeof(void **)); + for_each_object(p, s, addr, objects) + if (test_bit(slab_index(p, s, addr), map)) + vector[count++] = p; + + if (s->isolate) + private = s->isolate(s, vector, count); + else + /* + * Objects do not need to be isolated. + */ + private = NULL; + + /* + * Pinned the objects. Now we can drop the slab lock. The slab + * is frozen so it cannot vanish from under us nor will + * allocations be performed on the slab. However, unlocking the + * slab will allow concurrent slab_frees to proceed. So + * the subsystem must have a way to tell from the content + * of the object that it was freed. + * + * If neither RCU nor ctor is being used then the object + * may be modified by the allocator after being freed + * which may disrupt the ability of the migrate function + * to tell if the object is free or not. + */ + slab_unlock(page); + local_irq_restore(flags); + + /* + * Perform callbacks to move the objects. + */ + s->migrate(s, vector, count, node, private); +} + +/* + * Move slab objects on a particular node of the cache. + * Release slabs with zero objects and tryg to call the move function for + * slabs with less than the configured percentage of objects allocated. + * + * Returns the number of slabs left on the node after the operation. + */ +static unsigned long __move(struct kmem_cache *s, int node, + int target_node, int ratio) +{ + unsigned long flags; + struct page *page, *page2; + LIST_HEAD(move_list); + struct kmem_cache_node *n = get_node(s, node); + + if (node == target_node && n->nr_partial <= 1) + /* + * Trying to reduce fragmentataion on a node but there is + * only a single or no partial slab page. This is already + * the optimal object density that we can reach + */ + goto out; + + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &n->partial, lru) { + if (!slab_trylock(page)) + /* Busy slab. Get out of the way */ + continue; + + if (page->inuse) { + if (page->inuse > ratio * page->objects / 100) { + slab_unlock(page); + /* + * Skip slab because the object density + * in the slab page is high enough + */ + continue; + } + + list_move(&page->lru, &move_list); + if (s->migrate) { + /* Remove page from being considered for allocations */ + n->nr_partial--; + page->frozen = 1; + } + slab_unlock(page); + } else { + /* Empty slab page */ + list_del(&page->lru); + n->nr_partial--; + slab_unlock(page); + discard_slab(s, page); + } + } + + if (!s->migrate) + /* + * No defrag method. By simply putting the zaplist at the + * end of the partial list we can let them simmer longer + * and thus increase the chance of all objects being + * reclaimed. + * + * We have effectively sorted the partial list and put + * the slabs with more objects first. As soon as they + * are allocated they are going to be removed from the + * partial list. + */ + list_splice(&move_list, n->partial.prev); + + + spin_unlock_irqrestore(&n->list_lock, flags); + + if (s->migrate && !list_empty(&move_list)) { + void **scratch = alloc_scratch(); + struct page *page; + struct page *page2; + + if (scratch) { + /* Try to remove / move the objects left */ + list_for_each_entry(page, &move_list, lru) { + if (page->inuse) + kmem_cache_move(page, scratch, target_node); + } + kfree(scratch); + } + + /* Inspect results and dispose of pages */ + spin_lock_irqsave(&n->list_lock, flags); + list_for_each_entry_safe(page, page2, &move_list, lru) { + list_del(&page->lru); + slab_lock(page); + page->frozen = 0; + + if (page->inuse) { + /* + * Objects left in slab page move it to + * the tail of the partial list to + * increase the change that the freeing + * of the remaining objects will + * free the slab page + */ + n->nr_partial++; + list_add_tail(&page->lru, &n->partial); + slab_unlock(page); + + } else { + slab_unlock(page); + discard_slab(s, page); + } + } + spin_unlock_irqrestore(&n->list_lock, flags); + } +out: + return atomic_long_read(&n->nr_slabs); +} + +/* + * Defrag slabs conditional on the amount of fragmentation in a page. + */ +int kmem_cache_defrag(int node) +{ + struct kmem_cache *s; + unsigned long left = 0; + + /* + * kmem_cache_defrag may be called from the reclaim path which may be + * called for any page allocator alloc. So there is the danger that we + * get called in a situation where slub already acquired the slub_lock + * for other purposes. + */ + if (!mutex_trylock(&slab_mutex)) + return 0; + + list_for_each_entry(s, &slab_caches, list) { + /* + * Defragmentable caches come first. If the slab cache is not + * defragmentable then we can stop traversing the list. + */ + if (!s->migrate) + break; + + if (node == -1) { + int nid; + + for_each_node_state(nid, N_NORMAL_MEMORY) + if (s->node[nid]->nr_partial > MAX_PARTIAL) + left += __move(s, nid, nid, s->defrag_ratio); + } else + left += __move(s, node, node, 100); + + } + mutex_unlock(&slab_mutex); + return left; +} +EXPORT_SYMBOL(kmem_cache_defrag); + +/* + * kmem_cache_shrink reduces the memory footprint of a slab cache + * by as much as possible. This works by removing empty slabs from + * the partial list, migrating slab objects to denser slab pages + * (if the slab cache supports that) or reorganizing the partial + * list so that denser slab pages come first and less dense + * allocated slab pages are at the end. + */ +int __kmem_cache_shrink(struct kmem_cache *s) +{ + int node; + int left = 0; + + flush_all(s); + for_each_node_state(node, N_NORMAL_MEMORY) + left += __move(s, node, node, 100); + + return 0; +} +EXPORT_SYMBOL(__kmem_cache_shrink); + void kmem_cache_setup_mobility(struct kmem_cache *s, kmem_isolate_func isolate, kmem_migrate_func migrate) { int max_objects = oo_objects(s->max); /* - * Defragmentable slabs must have a ctor otherwise objects may be - * in an undetermined state after they are allocated. + * Mobile objects must have a ctor otherwise the + * object may be in an undefined state on allocation. + * + * Since the object may need to be inspected by the + * migration function at any time after allocation we + * must ensure that the object always has a defined + * state. */ BUG_ON(!s->ctor); From patchwork Thu Dec 20 19:22:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Christoph Lameter (Ampere)" X-Patchwork-Id: 10739437 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 050DA14E5 for ; Thu, 20 Dec 2018 19:22:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E600128CA1 for ; Thu, 20 Dec 2018 19:22:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D9D9828D22; Thu, 20 Dec 2018 19:22:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6FC8028CA1 for ; Thu, 20 Dec 2018 19:22:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DAC0C8E0013; Thu, 20 Dec 2018 14:22:03 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CB31D8E0001; Thu, 20 Dec 2018 14:22:03 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B367A8E0013; Thu, 20 Dec 2018 14:22:03 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id 8069E8E0001 for ; Thu, 20 Dec 2018 14:22:03 -0500 (EST) Received: by mail-qt1-f198.google.com with SMTP id 41so2902871qto.17 for ; Thu, 20 Dec 2018 11:22:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:subject:references:mime-version :content-disposition:feedback-id; bh=GqEKdOJa4NqL/PJEDbtnXVuUXZJi3QIuuXaA6btbLhU=; b=nHReTCOiqL3G6yYp7eWlO2T+A2vlrC10+skBDJsm0pv12mhQnppktoyjtDQp6Rk8zM WWlKgPfockxxpAwLQDgmpZjJobFU8BC2jzevtssSzX4nBL6HTHYECTL9r0FDjoRdGDJz BGFW6/u4ZZbWr8WArjjIo0Cwk8gbw4K3WWkMd+6FLho7XtiPbNU6K/Wrd80sZGr/nN3X tZkKOj09WGY9HPiES1l9EeoGgKs4ckPlvYBv+6iofqgXTd2XKOohD18jd5kIS4O6vmO1 7zJf3rNEX783EAP9wvQo0wr4x1O+ShOwfU0r7kG9m6QvYajYa17nGhipGqswCun3NKA5 tzEQ== X-Gm-Message-State: AA+aEWa3bnjyc42Ld5bJfyWPGIzdDEby3WFbGlj2RNRfvGScJW7TACW6 7bAqTt9aZ9ee84uisSFJqD2OZOyRHQ+VaAwpCpJY5jvZsNs0VBPnwXmCX1/Dy+NbKo2j9PZ2ZGQ AEzOMfysqkQ+fVPiwWJ1OfgQ89+U8QQdI8dFJGP736qKAvynwPjJV63x2Vxt8UT8= X-Received: by 2002:ae9:ed89:: with SMTP id c131mr25660818qkg.222.1545333723277; Thu, 20 Dec 2018 11:22:03 -0800 (PST) X-Google-Smtp-Source: AFSGD/WI62eyyVDJjUj9H6eiZtREibPLWBQmWzKIMSBDNbCMz6nwr/y9Q/68GW7haJBI8wB00yb+ X-Received: by 2002:ae9:ed89:: with SMTP id c131mr25660797qkg.222.1545333722816; Thu, 20 Dec 2018 11:22:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545333722; cv=none; d=google.com; s=arc-20160816; b=fYXuctYSgOla2BRAZ7fkB8vSCMst1lv/UlfSncTHm+zpa22uzdMVQJA9UTQh9o2ilM oMYSuXGnrNWcYCvHDm8BYUvrv5Q0bPMoXsa0Aq2tLfhSRksnsE8wuQZvPa7xf98gi/cp aC9DdaMe134FTN8i7L70kQ05v2XqkLnhFCoyYecMuJnNjYuVQkIoF69afxP6I6K12/Nj yGlcQeEtWL4ab2XBCCWKQWGMUL9rZi7cNns8x1mWZKAUQegDrkn31sthpGFIAfpd88hJ skBaMO6//e1O5CRtvg3iYWyxC2HSwPNqgIKmZxHAFH0Tm2/VeSl99UDP4LxIKLZpUvov aruw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-disposition:mime-version:references:subject:cc :cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:to:from:date:user-agent:message-id :dkim-signature; bh=GqEKdOJa4NqL/PJEDbtnXVuUXZJi3QIuuXaA6btbLhU=; b=v6mVioNm2bsl/veC5+XXckWZZXaDP1H4zBhcZjqWaV+BliDaVyh9XHKAUtbFJylb1C ESPhTz0kbwbzfjlPC2/qekYwUamYlfcdaioOEJu++USM72D+fNyP/g+SN24Dj6+p1dok WrxpWL/8XSktnGIdHWib1LmN1TAkKdpk1XHcNudUxaNjfIYyfe9ePajvzfElHXEdHZGK vfXcuEsP1aPqnnE6EQnFyjST3HGw/zU2OSCiKy271cuIFLUD5gURmx29/WFlN1Z+f0Zx HCu8ed79yK8HaZE6ZOSFG/ZUG4IVbarR9nt76PqJiMv/NKFxf3YeqZT1iYH1kry90gNv HHmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=G61d0PPs; spf=pass (google.com: domain of 01000167cd114cfc-077a81a2-a425-4578-a5af-89d000f59749-000000@amazonses.com designates 54.240.9.112 as permitted sender) smtp.mailfrom=01000167cd114cfc-077a81a2-a425-4578-a5af-89d000f59749-000000@amazonses.com Received: from a9-112.smtp-out.amazonses.com (a9-112.smtp-out.amazonses.com. [54.240.9.112]) by mx.google.com with ESMTPS id j62si2339122qkj.139.2018.12.20.11.22.02 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 20 Dec 2018 11:22:02 -0800 (PST) Received-SPF: pass (google.com: domain of 01000167cd114cfc-077a81a2-a425-4578-a5af-89d000f59749-000000@amazonses.com designates 54.240.9.112 as permitted sender) client-ip=54.240.9.112; Authentication-Results: mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=G61d0PPs; spf=pass (google.com: domain of 01000167cd114cfc-077a81a2-a425-4578-a5af-89d000f59749-000000@amazonses.com designates 54.240.9.112 as permitted sender) smtp.mailfrom=01000167cd114cfc-077a81a2-a425-4578-a5af-89d000f59749-000000@amazonses.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1545333722; h=Message-Id:Date:From:To:Cc:Cc:Cc:CC:Cc:Cc:Cc:Cc:Cc:Cc:Cc:Subject:References:MIME-Version:Content-Type:Feedback-ID; bh=BCm2PiyMf3tx6ZvzxPPkAP12ULxpZW06gS2AyjZcLp8=; b=G61d0PPsXs+Tscj8XOVELBwbPKNf0QVDMLAyIG79j+4VmVPBVzFJG5Dz0m/9PH2h ry86kMX1hOxMd6oDUO344KEEn9bQUqwhIDCnskTOtohaz/7Sf0sRWOibU7TT1d9AcwP bhouU10HazEhSJ9aeKk0WJh6roPvH8OD7Ou55ljE= Message-ID: <01000167cd114cfc-077a81a2-a425-4578-a5af-89d000f59749-000000@email.amazonses.com> User-Agent: quilt/0.65 Date: Thu, 20 Dec 2018 19:22:02 +0000 From: Christoph Lameter To: Matthew Wilcox Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: Pekka Enberg CC: akpm@linux-foundation.org Cc: Mel Gorman Cc: andi@firstfloor.org Cc: Rik van Riel Cc: Dave Chinner Cc: Christoph Hellwig Cc: Michal Hocko Cc: Mike Kravetz Subject: [RFC 6/7] slub: Extend slabinfo to support -D and -F options References: <20181220192145.023162076@linux.com> MIME-Version: 1.0 Content-Disposition: inline; filename=extend_slabinfo X-SES-Outgoing: 2018.12.20-54.240.9.112 Feedback-ID: 1.us-east-1.fQZZZ0Xtj2+TD7V5apTT/NrT6QKuPgzCT/IC7XYgDKI=:AmazonSES X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP -F lists caches that support moving and defragmentation -C lists caches that use a ctor. Change field names for defrag_ratio and remote_node_defrag_ratio. Add determination of the allocation ratio for a slab. The allocation ratio is the percentage of available slots for objects in use. Signed-off-by: Christoph Lameter --- Documentation/vm/slabinfo.c | 48 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 43 insertions(+), 5 deletions(-) Index: linux/tools/vm/slabinfo.c =================================================================== --- linux.orig/tools/vm/slabinfo.c +++ linux/tools/vm/slabinfo.c @@ -33,6 +33,8 @@ struct slabinfo { unsigned int hwcache_align, object_size, objs_per_slab; unsigned int sanity_checks, slab_size, store_user, trace; int order, poison, reclaim_account, red_zone; + int movable, ctor; + int defrag_ratio, remote_node_defrag_ratio; unsigned long partial, objects, slabs, objects_partial, objects_total; unsigned long alloc_fastpath, alloc_slowpath; unsigned long free_fastpath, free_slowpath; @@ -67,6 +69,8 @@ int show_report; int show_alias; int show_slab; int skip_zero = 1; +int show_movable; +int show_ctor; int show_numa; int show_track; int show_first_alias; @@ -109,14 +113,16 @@ static void fatal(const char *x, ...) static void usage(void) { - printf("slabinfo 4/15/2011. (c) 2007 sgi/(c) 2011 Linux Foundation.\n\n" - "slabinfo [-ahnpvtsz] [-d debugopts] [slab-regexp]\n" + printf("slabinfo 4/15/2017. (c) 2007 sgi/(c) 2011 Linux Foundation/(c) 2017 Jump Trading LLC.\n\n" + "slabinfo [-aCdDefFhnpvtsz] [-d debugopts] [slab-regexp]\n" "-a|--aliases Show aliases\n" "-A|--activity Most active slabs first\n" "-d|--debug= Set/Clear Debug options\n" + "-C|--ctor Show slabs with ctors\n" "-D|--display-active Switch line format to activity\n" "-e|--empty Show empty slabs\n" "-f|--first-alias Show first alias\n" + "-F|--movable Show caches that support movable objects\n" "-h|--help Show usage information\n" "-i|--inverted Inverted list\n" "-l|--slabs Show slabs\n" @@ -369,7 +375,7 @@ static void slab_numa(struct slabinfo *s return; if (!line) { - printf("\n%-21s:", mode ? "NUMA nodes" : "Slab"); + printf("\n%-21s: Rto ", mode ? "NUMA nodes" : "Slab"); for(node = 0; node <= highest_node; node++) printf(" %4d", node); printf("\n----------------------"); @@ -378,6 +384,7 @@ static void slab_numa(struct slabinfo *s printf("\n"); } printf("%-21s ", mode ? "All slabs" : s->name); + printf("%3d ", s->remote_node_defrag_ratio); for(node = 0; node <= highest_node; node++) { char b[20]; @@ -535,6 +542,8 @@ static void report(struct slabinfo *s) printf("** Slabs are destroyed via RCU\n"); if (s->reclaim_account) printf("** Reclaim accounting active\n"); + if (s->movable) + printf("** Defragmentation at %d%%\n", s->defrag_ratio); printf("\nSizes (bytes) Slabs Debug Memory\n"); printf("------------------------------------------------------------------------\n"); @@ -585,6 +594,12 @@ static void slabcache(struct slabinfo *s if (show_empty && s->slabs) return; + if (show_movable && !s->movable) + return; + + if (show_ctor && !s->ctor) + return; + if (sort_loss == 0) store_size(size_str, slab_size(s)); else @@ -599,6 +614,10 @@ static void slabcache(struct slabinfo *s *p++ = '*'; if (s->cache_dma) *p++ = 'd'; + if (s->movable) + *p++ = 'F'; + if (s->ctor) + *p++ = 'C'; if (s->hwcache_align) *p++ = 'A'; if (s->poison) @@ -633,7 +652,8 @@ static void slabcache(struct slabinfo *s printf("%-21s %8ld %7d %15s %14s %4d %1d %3ld %3ld %s\n", s->name, s->objects, s->object_size, size_str, dist_str, s->objs_per_slab, s->order, - s->slabs ? (s->partial * 100) / s->slabs : 100, + s->slabs ? (s->partial * 100) / + (s->slabs * s->objs_per_slab) : 100, s->slabs ? (s->objects * s->object_size * 100) / (s->slabs * (page_size << s->order)) : 100, flags); @@ -1252,7 +1272,17 @@ static void read_slab_dir(void) slab->cpu_partial_free = get_obj("cpu_partial_free"); slab->alloc_node_mismatch = get_obj("alloc_node_mismatch"); slab->deactivate_bypass = get_obj("deactivate_bypass"); + slab->defrag_ratio = get_obj("defrag_ratio"); + slab->remote_node_defrag_ratio = + get_obj("remote_node_defrag_ratio"); chdir(".."); + if (read_slab_obj(slab, "ops")) { + if (strstr(buffer, "ctor :")) + slab->ctor = 1; + if (strstr(buffer, "migrate :")) + slab->movable = 1; + } + if (slab->name[0] == ':') alias_targets++; slab++; @@ -1329,6 +1359,8 @@ static void xtotals(void) } struct option opts[] = { + { "ctor", no_argument, NULL, 'C' }, + { "movable", no_argument, NULL, 'F' }, { "aliases", no_argument, NULL, 'a' }, { "activity", no_argument, NULL, 'A' }, { "debug", optional_argument, NULL, 'd' }, @@ -1364,7 +1396,7 @@ int main(int argc, char *argv[]) page_size = getpagesize(); - while ((c = getopt_long(argc, argv, "aAd::Defhil1noprstvzTSN:LXBU", + while ((c = getopt_long(argc, argv, "aACd::DefFhil1noprstvzTSN:LXBU", opts, NULL)) != -1) switch (c) { case '1': @@ -1420,6 +1452,12 @@ int main(int argc, char *argv[]) case 'z': skip_zero = 0; break; + case 'C': + show_ctor = 1; + break; + case 'F': + show_movable = 1; + break; case 'T': show_totals = 1; break; From patchwork Thu Dec 20 19:22:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Christoph Lameter (Ampere)" X-Patchwork-Id: 10739439 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF02E746 for ; Thu, 20 Dec 2018 19:22:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9D7BD28CA1 for ; Thu, 20 Dec 2018 19:22:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9107428D22; Thu, 20 Dec 2018 19:22:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 29C0828CA1 for ; Thu, 20 Dec 2018 19:22:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A7998E0015; Thu, 20 Dec 2018 14:22:05 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 208938E0001; Thu, 20 Dec 2018 14:22:05 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A6BE8E0015; Thu, 20 Dec 2018 14:22:04 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by kanga.kvack.org (Postfix) with ESMTP id CFACB8E0001 for ; Thu, 20 Dec 2018 14:22:04 -0500 (EST) Received: by mail-qt1-f198.google.com with SMTP id k90so2981832qte.0 for ; Thu, 20 Dec 2018 11:22:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:message-id:user-agent:date:from :to:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:subject:references:mime-version :content-disposition:feedback-id; bh=w1fmM/A7/dLGujgt4qj/eIcE2IM8trq03+WjkP6rwKM=; b=T33Wz0r5W+e2s7P/AgDFB1dXKWpfRPqWvdZqnz6+sMy+j4quY3PR2uqb7SUNjbDUMK 3nkC9VmVMYy8CKk+br5wz3kF2QiTdPMxXpFCOfqghacXOqu8IVOxHXSUIeTQgNYW/ozA inVTOgB2GLDe7Q4JVr1/yvBL/5HwCQQZK/3LcPALvYF954AdhcTz0LOAQF0a6aIbe0id 2Jli6kqok47u5iKCVeLjNVp7LdHOYlBo03VfnQukWjZm1RbVTdj6xBsxsfqrmPu39Ulp GxKq28Nv7+KrN8jeCejlDyUysXJSgQv51VGik/sPJmq8oNSFaSoCfsTg2DZk2SPesaXD 73DA== X-Gm-Message-State: AA+aEWaW7/1XZR7IpYNVaK3pEMBsxGFc1lJz9lQwwZ8qvGgaLYXt2uRU GMKiRBl0897D7JxoOSHJGorx0a5ARd338SFanzKAFZH0nYOvIsanbfxr/a5sx6ANrHqJBMySmqQ zS78LXM1OPsKkEQO/Lu3y+UfzsFUgdyn+DoJdoHqREDG4R3IzP3v3xJhbSen004Y= X-Received: by 2002:ac8:518d:: with SMTP id c13mr26615120qtn.254.1545333724613; Thu, 20 Dec 2018 11:22:04 -0800 (PST) X-Google-Smtp-Source: AFSGD/VelJAVgYE52SpK3mSrDBa59Ay/Yl3QeC++v5OJ6ioEeEBN0h6sEsDeYKte3Hl80l/ZCX+c X-Received: by 2002:ac8:518d:: with SMTP id c13mr26615105qtn.254.1545333724269; Thu, 20 Dec 2018 11:22:04 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545333724; cv=none; d=google.com; s=arc-20160816; b=YdfSJZ7QRTpUZYyA8pzb4TqP/K86QiLe2mVtzbL7yNwlf9qEH/KBhhzivBEedTURTA JFPzdr1+yJn678svy8Cur4hytWrb1RGe2r5RZpCt9G0hTgaWwEVjthuw8+ZBDyrXgB12 wDtqve67y59tlJm662yAcEEmBX9XxMGYxGiPpaU2gbjMQfdwnpAEqgDYmtqdkWPC51jC 8ylWeYwt8wd1NBDGZQz8A8icpQOwYc8AMC2Bw4ogDWE+p2/hNJ0h4tdb4qe7zmaPAUZ4 ahQCxcIgq/7hDqQQak+gESavn92JIwir/da6Zm+5W0RRGykwy0QkEXXRUsD/vGWni0Y7 6S+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=feedback-id:content-disposition:mime-version:references:subject:cc :cc:cc:cc:cc:cc:cc:cc:cc:cc:cc:to:from:date:user-agent:message-id :dkim-signature; bh=w1fmM/A7/dLGujgt4qj/eIcE2IM8trq03+WjkP6rwKM=; b=K5QYwOpz7FOYaP6HqB+VsXIGzADkSln0uixUBGJy+A+MjstFJe7gU65mNxJrnns88R ocedsguk2D4j6Y8sPayXzclvqQQWK0gOwNe2fyNCK/vVbXyxlDMVi6QnY5hMc7ecRn+X f/Q/jz+JAyKvNfGG0cnfUumcUWptbiQgagyz51evn6th8J5MwuWSZd6RgQHaJWUcy1wq DPfjtPp+ENFwBcq1Bx38Pcs38NheW/RNZRjtkrR7/lKc/uxtIJVYR5qn0CyZbpK2lJkE cta+0KOm7MJx43LxOFJaI8CNUXXheKnYwpoiG88nWfTlejPaH6huYtnSmrn41G/0JOjq ++QA== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=SCvVL+ju; spf=pass (google.com: domain of 01000167cd11517f-f122b002-1a61-46c9-af1a-5c7cf01a397d-000000@amazonses.com designates 54.240.9.36 as permitted sender) smtp.mailfrom=01000167cd11517f-f122b002-1a61-46c9-af1a-5c7cf01a397d-000000@amazonses.com Received: from a9-36.smtp-out.amazonses.com (a9-36.smtp-out.amazonses.com. [54.240.9.36]) by mx.google.com with ESMTPS id l2si4247657qtj.22.2018.12.20.11.22.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 20 Dec 2018 11:22:04 -0800 (PST) Received-SPF: pass (google.com: domain of 01000167cd11517f-f122b002-1a61-46c9-af1a-5c7cf01a397d-000000@amazonses.com designates 54.240.9.36 as permitted sender) client-ip=54.240.9.36; Authentication-Results: mx.google.com; dkim=fail header.i=@amazonses.com header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=SCvVL+ju; spf=pass (google.com: domain of 01000167cd11517f-f122b002-1a61-46c9-af1a-5c7cf01a397d-000000@amazonses.com designates 54.240.9.36 as permitted sender) smtp.mailfrom=01000167cd11517f-f122b002-1a61-46c9-af1a-5c7cf01a397d-000000@amazonses.com DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug; d=amazonses.com; t=1545333723; h=Message-Id:Date:From:To:Cc:Cc:Cc:CC:Cc:Cc:Cc:Cc:Cc:Cc:Cc:Subject:References:MIME-Version:Content-Type:Feedback-ID; bh=qNdEXLooQmyJC8nAB/4MyBKqNpzM7jv1JHGkgte2EEs=; b=SCvVL+juPo8Ksu9+O0sk/r8AiUR8JGnyzXhP7mYdvc3cEQ0ulPEhnt+VLcPLe1mc smDDyDaYkQ6dcwgZE1Tk0m8tUHdSf90cH0en+w2zBJBvdu+bFjmIMne/tJlZiqGphRG 85bQcTIpVPEKivVme9DnvBzP1GZjcA05QzDioYdE= Message-ID: <01000167cd11517f-f122b002-1a61-46c9-af1a-5c7cf01a397d-000000@email.amazonses.com> User-Agent: quilt/0.65 Date: Thu, 20 Dec 2018 19:22:03 +0000 From: Christoph Lameter To: Matthew Wilcox Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: Pekka Enberg CC: akpm@linux-foundation.org Cc: Mel Gorman Cc: andi@firstfloor.org Cc: Rik van Riel Cc: Dave Chinner Cc: Christoph Hellwig Cc: Michal Hocko Cc: Mike Kravetz Subject: [RFC 7/7] xarray: Implement migration function for objects References: <20181220192145.023162076@linux.com> MIME-Version: 1.0 Content-Disposition: inline; filename=xarray X-SES-Outgoing: 2018.12.20-54.240.9.36 Feedback-ID: 1.us-east-1.fQZZZ0Xtj2+TD7V5apTT/NrT6QKuPgzCT/IC7XYgDKI=:AmazonSES X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Implement functions to migrate objects. This is based on initial code by Matthew Wilcox and was modified to work with slab object migration. Signed-off-by: Christoph Lameter Index: linux/lib/radix-tree.c =================================================================== --- linux.orig/lib/radix-tree.c +++ linux/lib/radix-tree.c @@ -1613,6 +1613,18 @@ static int radix_tree_cpu_dead(unsigned return 0; } + +extern void xa_object_migrate(void *tree_node, int numa_node); + +static void radix_tree_migrate(struct kmem_cache *s, void **objects, int nr, + int node, void *private) +{ + int i; + + for (i=0; iarray); + void __rcu **slot; + struct xa_node *new_node; + int i; + + /* Freed or not yet in tree then skip */ + if (!xa || xa == XA_FREE_MARK) + return; + + new_node = kmem_cache_alloc_node(radix_tree_node_cachep, GFP_KERNEL, numa_node); + + xa_lock_irq(xa); + + /* Check again..... */ + if (xa != node->array || !list_empty(&node->private_list)) { + node = new_node; + goto unlock; + } + + memcpy(new_node, node, sizeof(struct xa_node)); + + /* Move pointers to new node */ + INIT_LIST_HEAD(&new_node->private_list); + for (i = 0; i < XA_CHUNK_SIZE; i++) { + void *x = xa_entry_locked(xa, new_node, i); + + if (xa_is_node(x)) + rcu_assign_pointer(xa_to_node(x)->parent, new_node); + } + if (!new_node->parent) + slot = &xa->xa_head; + else + slot = &xa_parent_locked(xa, new_node)->slots[new_node->offset]; + rcu_assign_pointer(*slot, xa_mk_node(new_node)); + +unlock: + xa_unlock_irq(xa); + xa_node_free(node); + rcu_barrier(); + return; + +} + #ifdef XA_DEBUG void xa_dump_node(const struct xa_node *node) {