From patchwork Wed Apr 17 08:36:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Wool X-Patchwork-Id: 10904971 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8285C17E0 for ; Wed, 17 Apr 2019 08:36:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6782C28A56 for ; Wed, 17 Apr 2019 08:36:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5BADB28A93; Wed, 17 Apr 2019 08:36:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2C63428A56 for ; Wed, 17 Apr 2019 08:36:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 07DAA6B0007; Wed, 17 Apr 2019 04:36:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 006D46B000A; Wed, 17 Apr 2019 04:36:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEC1C6B000D; Wed, 17 Apr 2019 04:36:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lj1-f200.google.com (mail-lj1-f200.google.com [209.85.208.200]) by kanga.kvack.org (Postfix) with ESMTP id 6D52A6B0007 for ; Wed, 17 Apr 2019 04:36:38 -0400 (EDT) Received: by mail-lj1-f200.google.com with SMTP id o17so4744930ljd.2 for ; Wed, 17 Apr 2019 01:36:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:from:to:cc:subject :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=H8CKRiQBJrU9wnvoQVT3UbKuoETyr+5Jz1SZnQKUh/k=; b=d6bR7/ARk7/SSsjZeZ8IndUq/SkksrIsEdYlYvk9VNBCOYs+dJ1fS26L2mh3HRG6Ex wdrOg/HWr+XdgfIQG+AUwNL8oIwFdGL07Czon5rK5wB45NaMcRBq8wSG3HWW9K3Ab5lu oyPFbJwneU93p+7UqxrRf+iQz/bUvD6x4iW0QrG5phnZOmbHqcHDtOm2JCyTBPLfYa10 zS/4/pQs39KCRIBgO68Px5pSQN4GQV0F8CpPpxR2AT/feYFnUXFghGXYuFcaSUtLfLgx 35kOKWv6VmqCBEj0ux86913ePL+x7Fs5pjdurnvvLQofQrVFn2Vr8DkiOIOs2Bh3KGJu yB2Q== X-Gm-Message-State: APjAAAWshBD3utWmxkdFt6olWgq7eYrhpQam7sJECZxjnoc0fMBDUHUm jw6XO+/r4CQL87Pty0YFsu3w2eSYRHHH8Gau1/dR0aIEO4KjIMvv+Wp7qXVML5IAIs3qiv9RvKt HMRO2s67bNTtM9SF+DSf4hSiY4M5FimzT3gwiTzZJXyfquZrTvQtXaEk5YmnKfOmpoA== X-Received: by 2002:a2e:9e47:: with SMTP id g7mr8518801ljk.48.1555490197750; Wed, 17 Apr 2019 01:36:37 -0700 (PDT) X-Received: by 2002:a2e:9e47:: with SMTP id g7mr8518736ljk.48.1555490196173; Wed, 17 Apr 2019 01:36:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555490196; cv=none; d=google.com; s=arc-20160816; b=UHhr1c+miujERcFLb/DqmzJPm1eNJzrpzYYLC7D0Knn/BlJN0elBTVN56nvLMGf4Sb +hwarfsBeZh2lwUBZjVf2RQmCiTmUJoQnc3j0Ue9OrIyd9bgSPFFEL+mQfOQWt+Kzk5P HGHuEX2giUxUaszCfdjHBFU03DriLB1zooVPP+mns/mvKQ/m+Yme9hEEbKicLowFkKJ8 WVT8ve65BY3nRjn1KzqtOwX6N+twevYc0VD/yEfoI8sV/Iuypkus998ahDi8tcZ2jSW9 bsVYZArYn/PGF0AdbdO8REI4cl27VnXAALc9oZvplcDuOm+2SOhlUENX0nnUw49fQIq+ 8dpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=H8CKRiQBJrU9wnvoQVT3UbKuoETyr+5Jz1SZnQKUh/k=; b=ZutmdyJMN2gOb6JDktFcJ9LE1We6gWLUMnh7L/MtkTW1k1ICDS5YpqF/O97cYkfmU4 SF43h+FNYQ7Tnc/6oI8coUFSuM+u4vJ7oQQKBq0Znr5yyUUhiTl5/ZK7+tuJeMRYRTxj 3Y1lvHh25Ma5FvSEoZbZuUkUfpTj6/90kLmtn+0iNBz9bocEhN7oUTbQiczOVcWeNLCw 5l77iiMilLaSuyXbX7JOjp6onKM/OBZkY9cuv6TVyoDINmx0rsnHRbOCwmnJGjphbYfJ LZqpiO4WgvxFIyk4sZBoPS+dOp0BH6ZI0DTRDgqA4NRgip2QykqKl0qUiM4BYJCXTxYU j1dg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=R1IRcJSV; spf=pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=vitalywool@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id q15sor32994604lji.37.2019.04.17.01.36.36 for (Google Transport Security); Wed, 17 Apr 2019 01:36:36 -0700 (PDT) Received-SPF: pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=R1IRcJSV; spf=pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=vitalywool@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=H8CKRiQBJrU9wnvoQVT3UbKuoETyr+5Jz1SZnQKUh/k=; b=R1IRcJSVKlPeG1OzA5UM0heY9883GHBOCEJII5rnA43y+Hql2xG5lkLSjzFDtahSXu 1ma+PUr7KSIMLlypnMKJwmoTJpQJp9vo6zdW8YiJm2UnkCvK6N22CErV2gvJRKcxd/Bj rIvLA+VkZNpPRnlLUTIEsV/PlYEKIGOlIhZIUP3MBQ2vYt7AjaOTR90eCJKgGfEx3VkS LBFDFLbG02BvXQxDr6gp1GWZfMLZCsNwYM9jPRtQ4s+hJwCwBIKXJuGk5Oxm3ti7xwD3 nBJhv+1fKHAOUytEruKO9kvx4NMrBepjRLUU5GBOn7KaJgnkQSzl1n/vf64DEK3fqNKs M1UA== X-Google-Smtp-Source: APXvYqya60Kq55uGra+BVlxGzHEw6mm4ntwjQ+Ryeh+DdqnFhJIqgXVA4baUPQVnD5FBK8GEVLMgnQ== X-Received: by 2002:a2e:5d56:: with SMTP id r83mr46839339ljb.74.1555490195243; Wed, 17 Apr 2019 01:36:35 -0700 (PDT) Received: from seldlx21914.corpusers.net ([37.139.156.40]) by smtp.gmail.com with ESMTPSA id l13sm7773830ljj.96.2019.04.17.01.36.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Apr 2019 01:36:34 -0700 (PDT) Date: Wed, 17 Apr 2019 10:36:33 +0200 From: Vitaly Wool To: Linux-MM , linux-kernel@vger.kernel.org Cc: Dan Streetman , Andrew Morton , Oleksiy.Avramchenko@sony.com, Bartlomiej Zolnierkiewicz , Krzysztof Kozlowski Subject: [PATCHV2 1/4] z3fold: introduce helper functions Message-Id: <20190417103633.a4bb770b5bf0fb7e43ce1666@gmail.com> In-Reply-To: <20190417103510.36b055f3314e0e32b916b30a@gmail.com> References: <20190417103510.36b055f3314e0e32b916b30a@gmail.com> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.30; x86_64-unknown-linux-gnu) Mime-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP This patch introduces a separate helper function for object allocation, as well as 2 smaller helpers to add a buddy to the list and to get a pointer to the pool from the z3fold header. No functional changes here. Signed-off-by: Vitaly Wool --- mm/z3fold.c | 184 ++++++++++++++++++++++++++++------------------------ 1 file changed, 100 insertions(+), 84 deletions(-) diff --git a/mm/z3fold.c b/mm/z3fold.c index aee9b0b8d907..7a59875d880c 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -255,10 +255,15 @@ static enum buddy handle_to_buddy(unsigned long handle) return (handle - zhdr->first_num) & BUDDY_MASK; } +static inline struct z3fold_pool *zhdr_to_pool(struct z3fold_header *zhdr) +{ + return zhdr->pool; +} + static void __release_z3fold_page(struct z3fold_header *zhdr, bool locked) { struct page *page = virt_to_page(zhdr); - struct z3fold_pool *pool = zhdr->pool; + struct z3fold_pool *pool = zhdr_to_pool(zhdr); WARN_ON(!list_empty(&zhdr->buddy)); set_bit(PAGE_STALE, &page->private); @@ -295,9 +300,10 @@ static void release_z3fold_page_locked_list(struct kref *ref) { struct z3fold_header *zhdr = container_of(ref, struct z3fold_header, refcount); - spin_lock(&zhdr->pool->lock); + struct z3fold_pool *pool = zhdr_to_pool(zhdr); + spin_lock(&pool->lock); list_del_init(&zhdr->buddy); - spin_unlock(&zhdr->pool->lock); + spin_unlock(&pool->lock); WARN_ON(z3fold_page_trylock(zhdr)); __release_z3fold_page(zhdr, true); @@ -349,6 +355,23 @@ static int num_free_chunks(struct z3fold_header *zhdr) return nfree; } +/* Add to the appropriate unbuddied list */ +static inline void add_to_unbuddied(struct z3fold_pool *pool, + struct z3fold_header *zhdr) +{ + if (zhdr->first_chunks == 0 || zhdr->last_chunks == 0 || + zhdr->middle_chunks == 0) { + struct list_head *unbuddied = get_cpu_ptr(pool->unbuddied); + + int freechunks = num_free_chunks(zhdr); + spin_lock(&pool->lock); + list_add(&zhdr->buddy, &unbuddied[freechunks]); + spin_unlock(&pool->lock); + zhdr->cpu = smp_processor_id(); + put_cpu_ptr(pool->unbuddied); + } +} + static inline void *mchunk_memmove(struct z3fold_header *zhdr, unsigned short dst_chunk) { @@ -406,10 +429,8 @@ static int z3fold_compact_page(struct z3fold_header *zhdr) static void do_compact_page(struct z3fold_header *zhdr, bool locked) { - struct z3fold_pool *pool = zhdr->pool; + struct z3fold_pool *pool = zhdr_to_pool(zhdr); struct page *page; - struct list_head *unbuddied; - int fchunks; page = virt_to_page(zhdr); if (locked) @@ -430,18 +451,7 @@ static void do_compact_page(struct z3fold_header *zhdr, bool locked) } z3fold_compact_page(zhdr); - unbuddied = get_cpu_ptr(pool->unbuddied); - fchunks = num_free_chunks(zhdr); - if (fchunks < NCHUNKS && - (!zhdr->first_chunks || !zhdr->middle_chunks || - !zhdr->last_chunks)) { - /* the page's not completely free and it's unbuddied */ - spin_lock(&pool->lock); - list_add(&zhdr->buddy, &unbuddied[fchunks]); - spin_unlock(&pool->lock); - zhdr->cpu = smp_processor_id(); - } - put_cpu_ptr(pool->unbuddied); + add_to_unbuddied(pool, zhdr); z3fold_page_unlock(zhdr); } @@ -453,6 +463,67 @@ static void compact_page_work(struct work_struct *w) do_compact_page(zhdr, false); } +/* returns _locked_ z3fold page header or NULL */ +static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, + size_t size, bool can_sleep) +{ + struct z3fold_header *zhdr = NULL; + struct page *page; + struct list_head *unbuddied; + int chunks = size_to_chunks(size), i; + +lookup: + /* First, try to find an unbuddied z3fold page. */ + unbuddied = get_cpu_ptr(pool->unbuddied); + for_each_unbuddied_list(i, chunks) { + struct list_head *l = &unbuddied[i]; + + zhdr = list_first_entry_or_null(READ_ONCE(l), + struct z3fold_header, buddy); + + if (!zhdr) + continue; + + /* Re-check under lock. */ + spin_lock(&pool->lock); + l = &unbuddied[i]; + if (unlikely(zhdr != list_first_entry(READ_ONCE(l), + struct z3fold_header, buddy)) || + !z3fold_page_trylock(zhdr)) { + spin_unlock(&pool->lock); + zhdr = NULL; + put_cpu_ptr(pool->unbuddied); + if (can_sleep) + cond_resched(); + goto lookup; + } + list_del_init(&zhdr->buddy); + zhdr->cpu = -1; + spin_unlock(&pool->lock); + + page = virt_to_page(zhdr); + if (test_bit(NEEDS_COMPACTING, &page->private)) { + z3fold_page_unlock(zhdr); + zhdr = NULL; + put_cpu_ptr(pool->unbuddied); + if (can_sleep) + cond_resched(); + goto lookup; + } + + /* + * this page could not be removed from its unbuddied + * list while pool lock was held, and then we've taken + * page lock so kref_put could not be called before + * we got here, so it's safe to just call kref_get() + */ + kref_get(&zhdr->refcount); + break; + } + put_cpu_ptr(pool->unbuddied); + + return zhdr; +} /* * API Functions @@ -546,7 +617,7 @@ static void z3fold_destroy_pool(struct z3fold_pool *pool) static int z3fold_alloc(struct z3fold_pool *pool, size_t size, gfp_t gfp, unsigned long *handle) { - int chunks = 0, i, freechunks; + int chunks = size_to_chunks(size); struct z3fold_header *zhdr = NULL; struct page *page = NULL; enum buddy bud; @@ -561,56 +632,8 @@ static int z3fold_alloc(struct z3fold_pool *pool, size_t size, gfp_t gfp, if (size > PAGE_SIZE - ZHDR_SIZE_ALIGNED - CHUNK_SIZE) bud = HEADLESS; else { - struct list_head *unbuddied; - chunks = size_to_chunks(size); - -lookup: - /* First, try to find an unbuddied z3fold page. */ - unbuddied = get_cpu_ptr(pool->unbuddied); - for_each_unbuddied_list(i, chunks) { - struct list_head *l = &unbuddied[i]; - - zhdr = list_first_entry_or_null(READ_ONCE(l), - struct z3fold_header, buddy); - - if (!zhdr) - continue; - - /* Re-check under lock. */ - spin_lock(&pool->lock); - l = &unbuddied[i]; - if (unlikely(zhdr != list_first_entry(READ_ONCE(l), - struct z3fold_header, buddy)) || - !z3fold_page_trylock(zhdr)) { - spin_unlock(&pool->lock); - put_cpu_ptr(pool->unbuddied); - goto lookup; - } - list_del_init(&zhdr->buddy); - zhdr->cpu = -1; - spin_unlock(&pool->lock); - - page = virt_to_page(zhdr); - if (test_bit(NEEDS_COMPACTING, &page->private)) { - z3fold_page_unlock(zhdr); - zhdr = NULL; - put_cpu_ptr(pool->unbuddied); - if (can_sleep) - cond_resched(); - goto lookup; - } - - /* - * this page could not be removed from its unbuddied - * list while pool lock was held, and then we've taken - * page lock so kref_put could not be called before - * we got here, so it's safe to just call kref_get() - */ - kref_get(&zhdr->refcount); - break; - } - put_cpu_ptr(pool->unbuddied); - +retry: + zhdr = __z3fold_alloc(pool, size, can_sleep); if (zhdr) { if (zhdr->first_chunks == 0) { if (zhdr->middle_chunks != 0 && @@ -630,8 +653,9 @@ static int z3fold_alloc(struct z3fold_pool *pool, size_t size, gfp_t gfp, z3fold_page_unlock(zhdr); pr_err("No free chunks in unbuddied\n"); WARN_ON(1); - goto lookup; + goto retry; } + page = virt_to_page(zhdr); goto found; } bud = FIRST; @@ -662,8 +686,12 @@ static int z3fold_alloc(struct z3fold_pool *pool, size_t size, gfp_t gfp, if (!page) return -ENOMEM; - atomic64_inc(&pool->pages_nr); zhdr = init_z3fold_page(page, pool); + if (!zhdr) { + __free_page(page); + return -ENOMEM; + } + atomic64_inc(&pool->pages_nr); if (bud == HEADLESS) { set_bit(PAGE_HEADLESS, &page->private); @@ -680,19 +708,7 @@ static int z3fold_alloc(struct z3fold_pool *pool, size_t size, gfp_t gfp, zhdr->middle_chunks = chunks; zhdr->start_middle = zhdr->first_chunks + ZHDR_CHUNKS; } - - if (zhdr->first_chunks == 0 || zhdr->last_chunks == 0 || - zhdr->middle_chunks == 0) { - struct list_head *unbuddied = get_cpu_ptr(pool->unbuddied); - - /* Add to unbuddied list */ - freechunks = num_free_chunks(zhdr); - spin_lock(&pool->lock); - list_add(&zhdr->buddy, &unbuddied[freechunks]); - spin_unlock(&pool->lock); - zhdr->cpu = smp_processor_id(); - put_cpu_ptr(pool->unbuddied); - } + add_to_unbuddied(pool, zhdr); headless: spin_lock(&pool->lock); From patchwork Wed Apr 17 08:37:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Wool X-Patchwork-Id: 10904973 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED97B13B5 for ; Wed, 17 Apr 2019 08:37:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3EC928733 for ; Wed, 17 Apr 2019 08:37:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C84A628A95; Wed, 17 Apr 2019 08:37:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6B43F28733 for ; Wed, 17 Apr 2019 08:37:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8E4A86B0008; Wed, 17 Apr 2019 04:37:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 868CA6B000A; Wed, 17 Apr 2019 04:37:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70E046B000D; Wed, 17 Apr 2019 04:37:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by kanga.kvack.org (Postfix) with ESMTP id 052BD6B0008 for ; Wed, 17 Apr 2019 04:37:38 -0400 (EDT) Received: by mail-lf1-f71.google.com with SMTP id h25so3344569lfc.18 for ; Wed, 17 Apr 2019 01:37:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:from:to:cc:subject :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=eracmwt7PPl4AZZBoOATyO/9W4GSvbRyS4J/dbiNmIA=; b=dfeo8YOu8JjnJy9clvCWn7qfnWgfhPDYSBPAI45324J0caWc7FKO2e0Pox5OF7XtQS 16aHYXqobfZ1d0s/+rS09X3tp0UB6BTXGskEZ7YReC3YafR9L1dNB+0xxmFaLKYJ2Rme sYTetJTFesUl1Hvz1JS5eiMwiQUAvn4rL6PNyAzGtAcLf1+WJ07BML/ekooqzR9rICkX poHukCqs2MdfJu5UcAtBAn2EzYqdN2qgBQ8RUwC5occayaY3QNWEFDWsmpw7rw9CNzho COmc/HdHaJ6LAiocwguOLu157afrLcvit5i5/Z0tXMp27QrMwd3w0yArM8RO6stIaE7F fazw== X-Gm-Message-State: APjAAAW/mO8qw+BuzhhGT0eY1omTzFCs9sopCTtnk7Aaxpv1CwfoFw2/ gJtaOuVS+piSGgYSxCV1xad+qe1ODrksv8dRm20RVtrOdrMXAeUSFBsr9r2JEljP2de4xEJFrcw WzMcHXCoiovaOSbgaQ1lyp88y2A8UTaU5bV1xtkZ73FnAtWPqE0qKEIxltHwPpGuZYw== X-Received: by 2002:a2e:7f13:: with SMTP id a19mr46417703ljd.35.1555490257346; Wed, 17 Apr 2019 01:37:37 -0700 (PDT) X-Received: by 2002:a2e:7f13:: with SMTP id a19mr46417621ljd.35.1555490255996; Wed, 17 Apr 2019 01:37:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555490255; cv=none; d=google.com; s=arc-20160816; b=v24oTk7NMzSEAuKkdeOzjfYqGKGCRJOHf0+T7igodVy6p8BLWiUhzt2Axu4qJUaYMB 0Z60ljYQc69BYVUGFoucMJW57LwMi1AFi7/AV/liWtjzPd8gafb0lzfbhJcMJA/KQ3VQ 9OqraK4ubnQ9R9yzimxrppwxlzM9C3PGmtkA1fTHjHVVaFY1ucI3VuyWaNCw9ZYQ3/i5 sEwnIp5kR7CYdtBhBYaoXULT1v8P55oSwYVEJZn3VouNJOqhdH+qBotixKBwI66hiW0W fXCEo8zSG+YWIbKy7jMvNXuHtipr8KmvRFbXxPtcj23x0bf/MPaaI1PLSm+Rkm24Lo5Z xTdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=eracmwt7PPl4AZZBoOATyO/9W4GSvbRyS4J/dbiNmIA=; b=JRQNofssgiz3dnCs+rqI6bdCejLujQGVcN+pxe6Tjj81+AO8Fz3UnA3jai1t7flyFA OiGaQRwfMQM+2aMGnLG+3c6ZTGCUXHK7JVm5asNcU42k0qlAIhPgiTDcwcexqfz+pzLX cNipSeSm74DzgPTQCNlhLZI3J1J1kP685YelN2t/F85GUa6EqJlnKB2FCGV/qIKRL7jG 4/sqUbslzmKo+aEvTR7UEX934/2WGfnJZOq+vLPBdy24ipTd3fwY1/vZNTgAsaNgrJJx 70sFC4H371XamIZHDaWmayP0NlokwIhcuXhtkvHePh1AoqEkt2ob0OF6QN13uls/EunN gfMw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=tXGc0NZh; spf=pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=vitalywool@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id m3sor14750621lfp.45.2019.04.17.01.37.35 for (Google Transport Security); Wed, 17 Apr 2019 01:37:35 -0700 (PDT) Received-SPF: pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=tXGc0NZh; spf=pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=vitalywool@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=eracmwt7PPl4AZZBoOATyO/9W4GSvbRyS4J/dbiNmIA=; b=tXGc0NZhjaZsco2x643feyLYfETk7tP69DmsmI1ukNlFcsqVxOZPgNTDgtFTsGpGei zDp0VmDlpdSIaiqE58MDXrAFrLOcsd2ZgUb40kf8v9PBhD3bI8ob/4WqnZcnPTB8Wjts D7UFfdJOdF4zoWWmRJgX3/VMH+om5tlrkb9WVaRuRnzgsLKBZ0zjusQF991pOWXNXSdV sUz8cyhwdpR8nh6Ub/i3hfNLgjfq3zCYmsmTmXt2YZ7it6U5BtPhVIOCvnYGYbn5Ga+c 271dvA3vLMLYL7XdQfGgABhH7twTe7Z7PNeLa/lsXSITTNjU0Bnh7AWc/ZTyEmfCoxDY AysQ== X-Google-Smtp-Source: APXvYqx4ExKeIMbFg9T9tX6SZdDS0Cbp3kgvlbWCD26SsYSpLy2UMegfsoPIlI9gnvs8kofXCRRpow== X-Received: by 2002:ac2:43d8:: with SMTP id u24mr23190693lfl.94.1555490255103; Wed, 17 Apr 2019 01:37:35 -0700 (PDT) Received: from seldlx21914.corpusers.net ([37.139.156.40]) by smtp.gmail.com with ESMTPSA id d192sm1649933lfg.79.2019.04.17.01.37.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Apr 2019 01:37:34 -0700 (PDT) Date: Wed, 17 Apr 2019 10:37:33 +0200 From: Vitaly Wool To: Linux-MM , linux-kernel@vger.kernel.org Cc: Dan Streetman , Andrew Morton , Oleksiy.Avramchenko@sony.com, Bartlomiej Zolnierkiewicz , Krzysztof Kozlowski Subject: [PATCHv2 2/4] z3fold: improve compression by extending search Message-Id: <20190417103733.72ae81abe1552397c95a008e@gmail.com> In-Reply-To: <20190417103510.36b055f3314e0e32b916b30a@gmail.com> References: <20190417103510.36b055f3314e0e32b916b30a@gmail.com> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.30; x86_64-unknown-linux-gnu) Mime-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The current z3fold implementation only searches this CPU's page lists for a fitting page to put a new object into. This patch adds quick search for very well fitting pages (i. e. those having exactly the required number of free space) on other CPUs too, before allocating a new page for that object. Signed-off-by: Vitaly Wool --- mm/z3fold.c | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/mm/z3fold.c b/mm/z3fold.c index 7a59875d880c..29a4f1249bef 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -522,6 +522,42 @@ static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, } put_cpu_ptr(pool->unbuddied); + if (!zhdr) { + int cpu; + + /* look for _exact_ match on other cpus' lists */ + for_each_online_cpu(cpu) { + struct list_head *l; + + unbuddied = per_cpu_ptr(pool->unbuddied, cpu); + spin_lock(&pool->lock); + l = &unbuddied[chunks]; + + zhdr = list_first_entry_or_null(READ_ONCE(l), + struct z3fold_header, buddy); + + if (!zhdr || !z3fold_page_trylock(zhdr)) { + spin_unlock(&pool->lock); + zhdr = NULL; + continue; + } + list_del_init(&zhdr->buddy); + zhdr->cpu = -1; + spin_unlock(&pool->lock); + + page = virt_to_page(zhdr); + if (test_bit(NEEDS_COMPACTING, &page->private)) { + z3fold_page_unlock(zhdr); + zhdr = NULL; + if (can_sleep) + cond_resched(); + continue; + } + kref_get(&zhdr->refcount); + break; + } + } + return zhdr; } From patchwork Wed Apr 17 08:38:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Wool X-Patchwork-Id: 10904975 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 55DAC17E0 for ; Wed, 17 Apr 2019 08:38:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 38AB428A95 for ; Wed, 17 Apr 2019 08:38:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2B88428A96; Wed, 17 Apr 2019 08:38:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 39CC328A95 for ; Wed, 17 Apr 2019 08:38:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 561216B0007; Wed, 17 Apr 2019 04:38:32 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 50D076B000A; Wed, 17 Apr 2019 04:38:32 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3890D6B000D; Wed, 17 Apr 2019 04:38:32 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by kanga.kvack.org (Postfix) with ESMTP id BD3606B0007 for ; Wed, 17 Apr 2019 04:38:31 -0400 (EDT) Received: by mail-lf1-f71.google.com with SMTP id d17so2496449lfb.13 for ; Wed, 17 Apr 2019 01:38:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:from:to:cc:subject :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=Sd3LnrKhw2tgRz9UNxG9ueVkl/OGNG8X97V5bhmMYJo=; b=Mx5KdEfBVrp7gFYupuCGlh6D31P0YvjYds2Guo5tXEJ1TFRotcGYD76D3FelVWzCzJ HrAzEnCZast2ZjgGf9LU5UF8ypqE0+EYVsflZc4E3d/XoBgUMxfv2OfEqURH+7QGjAMt 9ShCvf2bl8GM1zA70wOYe9Z/XicOegfhP65kF0nXMGgZ1Tj78mHDGAaOiJEktv8lhFx7 afZqHl6xwdgtNDV4YLxOJiJgYJzlWnbCzsvEy4YVgKYcqBy1/+QRU8LkWjBv7s6WLoNs jZkWVjKd87dSjA/UEYV54/dpRXXmOGPzJd/u0GRvvekK5kkxDd5eoQ7ZMQ+KMxvrW08o JYmA== X-Gm-Message-State: APjAAAUw9uj+bUWzRYBll9gZNyBReFxleuyQy7cGm1GsBLT1iJ1rgQZf LHNfmhxq0ZUkjgneODhTRBXWclraC0bCduaRE2CVy3xPr/geoUIrbw1KLAIGgfCg9Fe++iKBnLf 4IvvecovCT5EiieAEWHX5UXrbrToGmrAcwY4rtEZRKMn9mo/JnxhPEJM1Qd2Do1UBkw== X-Received: by 2002:a2e:7d19:: with SMTP id y25mr24778086ljc.126.1555490311069; Wed, 17 Apr 2019 01:38:31 -0700 (PDT) X-Received: by 2002:a2e:7d19:: with SMTP id y25mr24777995ljc.126.1555490309227; Wed, 17 Apr 2019 01:38:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555490309; cv=none; d=google.com; s=arc-20160816; b=OT8IO/pG7SDGawQbIwnDQf9xCwGRui+2wFmhojN/nUdkKboSdC8cm5qK0O9MJB78H1 N/3gT6nZaQtYfcRrGExkVFMA8nJ3I4RtACq71Nw6rmo/fQWr/bK62mImud/t02VgLtks EW/Olcvj316UDvpK4pqrmEBLx8yqColmjtj05PZo/+HcM6AGPeCN2+iimrzlatXoV8Hu TjHeqRUak71qsfihtt2cBrhatRWyTAEFDzy5s40dUEEKpsvcJv4BdnNbgCOUZF8YBSUS w1rtvcv2707A6UTrje5swPZplYOmSlX6raAy/iNevAZ0Drv1a/i9YibYT9QOgoLEwqlM 2QQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=Sd3LnrKhw2tgRz9UNxG9ueVkl/OGNG8X97V5bhmMYJo=; b=LsdjgvpSkI3RL6TGir53EkrSKc5Xm5mlowKUafLorrEQ/P1z+Dl1mKsPG087i13q23 4QvH2jO6FmDT4XzTCYzyS7O1gFmmKJv12a7GjAw4hyLr4ahwDGJcUaeeMHrb47dUBESH an6NhyNzSj5P25v9pkwAZzOes6nBmzSNwqOIbI1GWZlE0rpp5d7C5NEiVLn6fMJKt8qq 3TvCPO+eCwtDDhzIEdzukXvfAsjNmwV86EQ+GwtvFQrLJKYyR3WfWB4OuRt3J25jmap9 PPyYuu9DSJtm+56WWSiP0/2Ak0gxW0F6eqtUBPpfjf0MixbTUsSAHfQWvNUuUaqLtC2+ oLbg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=rFGzCV6f; spf=pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=vitalywool@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id p13sor34197721ljj.25.2019.04.17.01.38.29 for (Google Transport Security); Wed, 17 Apr 2019 01:38:29 -0700 (PDT) Received-SPF: pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=rFGzCV6f; spf=pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=vitalywool@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Sd3LnrKhw2tgRz9UNxG9ueVkl/OGNG8X97V5bhmMYJo=; b=rFGzCV6fX6StJ5G0K5fC7EVFFfqaK1iDfsJPnIQ/7XkvmUDWe/a9Kb9Z9b2A8frmHm oqz9FpY0ZyPWyRTInMckiv6mrmYCgLVMRpf6qSYQwEZkhnkj0zt/90Afqtz7cGKzlAFp j4nbDNzAU5vsLKq6LkhFrALAsSu3o5+NipaRXx3jXYPbCSQ55TH2Ih5Wm0jGnUwDA0tF +I0LfLR8Xr6bAz7TGBiJx+Uo34S+OJ5z0L9u27l965felx/GMAabda5uLYKA08DUh42Q QI7wkN/WmynijmlAOf/zzY99WFgq/XnKS8o1sH8+uy9KAJ3Xx/peXr5cryIBef0z25Au 185w== X-Google-Smtp-Source: APXvYqzDOdmW2Tjh3gHF4VHAQw0qCWki0/5SCGLAS0g65kd+ISDVNM897NcFG+QvgPceqJZODVjA6g== X-Received: by 2002:a2e:3010:: with SMTP id w16mr46578193ljw.62.1555490308302; Wed, 17 Apr 2019 01:38:28 -0700 (PDT) Received: from seldlx21914.corpusers.net ([37.139.156.40]) by smtp.gmail.com with ESMTPSA id r2sm10796523ljd.16.2019.04.17.01.38.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Apr 2019 01:38:27 -0700 (PDT) Date: Wed, 17 Apr 2019 10:38:26 +0200 From: Vitaly Wool To: Linux-MM , linux-kernel@vger.kernel.org Cc: Dan Streetman , Andrew Morton , Oleksiy.Avramchenko@sony.com, Bartlomiej Zolnierkiewicz , Krzysztof Kozlowski Subject: [PATCHv2 3/4] z3fold: add structure for buddy handles Message-Id: <20190417103826.484eaf18c1294d682769880f@gmail.com> In-Reply-To: <20190417103510.36b055f3314e0e32b916b30a@gmail.com> References: <20190417103510.36b055f3314e0e32b916b30a@gmail.com> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.30; x86_64-unknown-linux-gnu) Mime-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP For z3fold to be able to move its pages per request of the memory subsystem, it should not use direct object addresses in handles. Instead, it will create abstract handles (3 per page) which will contain pointers to z3fold objects. Thus, it will be possible to change these pointers when z3fold page is moved. Signed-off-by: Vitaly Wool --- mm/z3fold.c | 185 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 145 insertions(+), 40 deletions(-) diff --git a/mm/z3fold.c b/mm/z3fold.c index 29a4f1249bef..bebc10083f1c 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -34,6 +34,29 @@ #include #include +/* + * NCHUNKS_ORDER determines the internal allocation granularity, effectively + * adjusting internal fragmentation. It also determines the number of + * freelists maintained in each pool. NCHUNKS_ORDER of 6 means that the + * allocation granularity will be in chunks of size PAGE_SIZE/64. Some chunks + * in the beginning of an allocated page are occupied by z3fold header, so + * NCHUNKS will be calculated to 63 (or 62 in case CONFIG_DEBUG_SPINLOCK=y), + * which shows the max number of free chunks in z3fold page, also there will + * be 63, or 62, respectively, freelists per pool. + */ +#define NCHUNKS_ORDER 6 + +#define CHUNK_SHIFT (PAGE_SHIFT - NCHUNKS_ORDER) +#define CHUNK_SIZE (1 << CHUNK_SHIFT) +#define ZHDR_SIZE_ALIGNED round_up(sizeof(struct z3fold_header), CHUNK_SIZE) +#define ZHDR_CHUNKS (ZHDR_SIZE_ALIGNED >> CHUNK_SHIFT) +#define TOTAL_CHUNKS (PAGE_SIZE >> CHUNK_SHIFT) +#define NCHUNKS ((PAGE_SIZE - ZHDR_SIZE_ALIGNED) >> CHUNK_SHIFT) + +#define BUDDY_MASK (0x3) +#define BUDDY_SHIFT 2 +#define SLOTS_ALIGN (0x40) + /***************** * Structures *****************/ @@ -47,9 +70,19 @@ enum buddy { FIRST, MIDDLE, LAST, - BUDDIES_MAX + BUDDIES_MAX = LAST }; +struct z3fold_buddy_slots { + /* + * we are using BUDDY_MASK in handle_to_buddy etc. so there should + * be enough slots to hold all possible variants + */ + unsigned long slot[BUDDY_MASK + 1]; + unsigned long pool; /* back link + flags */ +}; +#define HANDLE_FLAG_MASK (0x03) + /* * struct z3fold_header - z3fold page metadata occupying first chunks of each * z3fold page, except for HEADLESS pages @@ -58,7 +91,7 @@ enum buddy { * @page_lock: per-page lock * @refcount: reference count for the z3fold page * @work: work_struct for page layout optimization - * @pool: pointer to the pool which this page belongs to + * @slots: pointer to the structure holding buddy slots * @cpu: CPU which this page "belongs" to * @first_chunks: the size of the first buddy in chunks, 0 if free * @middle_chunks: the size of the middle buddy in chunks, 0 if free @@ -70,7 +103,7 @@ struct z3fold_header { spinlock_t page_lock; struct kref refcount; struct work_struct work; - struct z3fold_pool *pool; + struct z3fold_buddy_slots *slots; short cpu; unsigned short first_chunks; unsigned short middle_chunks; @@ -79,28 +112,6 @@ struct z3fold_header { unsigned short first_num:2; }; -/* - * NCHUNKS_ORDER determines the internal allocation granularity, effectively - * adjusting internal fragmentation. It also determines the number of - * freelists maintained in each pool. NCHUNKS_ORDER of 6 means that the - * allocation granularity will be in chunks of size PAGE_SIZE/64. Some chunks - * in the beginning of an allocated page are occupied by z3fold header, so - * NCHUNKS will be calculated to 63 (or 62 in case CONFIG_DEBUG_SPINLOCK=y), - * which shows the max number of free chunks in z3fold page, also there will - * be 63, or 62, respectively, freelists per pool. - */ -#define NCHUNKS_ORDER 6 - -#define CHUNK_SHIFT (PAGE_SHIFT - NCHUNKS_ORDER) -#define CHUNK_SIZE (1 << CHUNK_SHIFT) -#define ZHDR_SIZE_ALIGNED round_up(sizeof(struct z3fold_header), CHUNK_SIZE) -#define ZHDR_CHUNKS (ZHDR_SIZE_ALIGNED >> CHUNK_SHIFT) -#define TOTAL_CHUNKS (PAGE_SIZE >> CHUNK_SHIFT) -#define NCHUNKS ((PAGE_SIZE - ZHDR_SIZE_ALIGNED) >> CHUNK_SHIFT) - -#define BUDDY_MASK (0x3) -#define BUDDY_SHIFT 2 - /** * struct z3fold_pool - stores metadata for each z3fold pool * @name: pool name @@ -113,6 +124,7 @@ struct z3fold_header { * added buddy. * @stale: list of pages marked for freeing * @pages_nr: number of z3fold pages in the pool. + * @c_handle: cache for z3fold_buddy_slots allocation * @ops: pointer to a structure of user defined operations specified at * pool creation time. * @compact_wq: workqueue for page layout background optimization @@ -130,6 +142,7 @@ struct z3fold_pool { struct list_head lru; struct list_head stale; atomic64_t pages_nr; + struct kmem_cache *c_handle; const struct z3fold_ops *ops; struct zpool *zpool; const struct zpool_ops *zpool_ops; @@ -164,11 +177,65 @@ static int size_to_chunks(size_t size) static void compact_page_work(struct work_struct *w); +static inline struct z3fold_buddy_slots *alloc_slots(struct z3fold_pool *pool) +{ + struct z3fold_buddy_slots *slots = kmem_cache_alloc(pool->c_handle, + GFP_KERNEL); + + if (slots) { + memset(slots->slot, 0, sizeof(slots->slot)); + slots->pool = (unsigned long)pool; + } + + return slots; +} + +static inline struct z3fold_pool *slots_to_pool(struct z3fold_buddy_slots *s) +{ + return (struct z3fold_pool *)(s->pool & ~HANDLE_FLAG_MASK); +} + +static inline struct z3fold_buddy_slots *handle_to_slots(unsigned long handle) +{ + return (struct z3fold_buddy_slots *)(handle & ~(SLOTS_ALIGN - 1)); +} + +static inline void free_handle(unsigned long handle) +{ + struct z3fold_buddy_slots *slots; + int i; + bool is_free; + + if (handle & (1 << PAGE_HEADLESS)) + return; + + WARN_ON(*(unsigned long *)handle == 0); + *(unsigned long *)handle = 0; + slots = handle_to_slots(handle); + is_free = true; + for (i = 0; i <= BUDDY_MASK; i++) { + if (slots->slot[i]) { + is_free = false; + break; + } + } + + if (is_free) { + struct z3fold_pool *pool = slots_to_pool(slots); + + kmem_cache_free(pool->c_handle, slots); + } +} + /* Initializes the z3fold header of a newly allocated z3fold page */ static struct z3fold_header *init_z3fold_page(struct page *page, struct z3fold_pool *pool) { struct z3fold_header *zhdr = page_address(page); + struct z3fold_buddy_slots *slots = alloc_slots(pool); + + if (!slots) + return NULL; INIT_LIST_HEAD(&page->lru); clear_bit(PAGE_HEADLESS, &page->private); @@ -185,7 +252,7 @@ static struct z3fold_header *init_z3fold_page(struct page *page, zhdr->first_num = 0; zhdr->start_middle = 0; zhdr->cpu = -1; - zhdr->pool = pool; + zhdr->slots = slots; INIT_LIST_HEAD(&zhdr->buddy); INIT_WORK(&zhdr->work, compact_page_work); return zhdr; @@ -215,33 +282,57 @@ static inline void z3fold_page_unlock(struct z3fold_header *zhdr) spin_unlock(&zhdr->page_lock); } +/* Helper function to build the index */ +static inline int __idx(struct z3fold_header *zhdr, enum buddy bud) +{ + return (bud + zhdr->first_num) & BUDDY_MASK; +} + /* * Encodes the handle of a particular buddy within a z3fold page * Pool lock should be held as this function accesses first_num */ static unsigned long encode_handle(struct z3fold_header *zhdr, enum buddy bud) { - unsigned long handle; + struct z3fold_buddy_slots *slots; + unsigned long h = (unsigned long)zhdr; + int idx = 0; - handle = (unsigned long)zhdr; - if (bud != HEADLESS) { - handle |= (bud + zhdr->first_num) & BUDDY_MASK; - if (bud == LAST) - handle |= (zhdr->last_chunks << BUDDY_SHIFT); - } - return handle; + /* + * For a headless page, its handle is its pointer with the extra + * PAGE_HEADLESS bit set + */ + if (bud == HEADLESS) + return h | (1 << PAGE_HEADLESS); + + /* otherwise, return pointer to encoded handle */ + idx = __idx(zhdr, bud); + h += idx; + if (bud == LAST) + h |= (zhdr->last_chunks << BUDDY_SHIFT); + + slots = zhdr->slots; + slots->slot[idx] = h; + return (unsigned long)&slots->slot[idx]; } /* Returns the z3fold page where a given handle is stored */ -static struct z3fold_header *handle_to_z3fold_header(unsigned long handle) +static inline struct z3fold_header *handle_to_z3fold_header(unsigned long handle) { - return (struct z3fold_header *)(handle & PAGE_MASK); + unsigned long addr = handle; + + if (!(addr & (1 << PAGE_HEADLESS))) + addr = *(unsigned long *)handle; + + return (struct z3fold_header *)(addr & PAGE_MASK); } /* only for LAST bud, returns zero otherwise */ static unsigned short handle_to_chunks(unsigned long handle) { - return (handle & ~PAGE_MASK) >> BUDDY_SHIFT; + unsigned long addr = *(unsigned long *)handle; + + return (addr & ~PAGE_MASK) >> BUDDY_SHIFT; } /* @@ -251,13 +342,18 @@ static unsigned short handle_to_chunks(unsigned long handle) */ static enum buddy handle_to_buddy(unsigned long handle) { - struct z3fold_header *zhdr = handle_to_z3fold_header(handle); - return (handle - zhdr->first_num) & BUDDY_MASK; + struct z3fold_header *zhdr; + unsigned long addr; + + WARN_ON(handle & (1 << PAGE_HEADLESS)); + addr = *(unsigned long *)handle; + zhdr = (struct z3fold_header *)(addr & PAGE_MASK); + return (addr - zhdr->first_num) & BUDDY_MASK; } static inline struct z3fold_pool *zhdr_to_pool(struct z3fold_header *zhdr) { - return zhdr->pool; + return slots_to_pool(zhdr->slots); } static void __release_z3fold_page(struct z3fold_header *zhdr, bool locked) @@ -583,6 +679,11 @@ static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, pool = kzalloc(sizeof(struct z3fold_pool), gfp); if (!pool) goto out; + pool->c_handle = kmem_cache_create("z3fold_handle", + sizeof(struct z3fold_buddy_slots), + SLOTS_ALIGN, 0, NULL); + if (!pool->c_handle) + goto out_c; spin_lock_init(&pool->lock); spin_lock_init(&pool->stale_lock); pool->unbuddied = __alloc_percpu(sizeof(struct list_head)*NCHUNKS, 2); @@ -613,6 +714,8 @@ static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, out_unbuddied: free_percpu(pool->unbuddied); out_pool: + kmem_cache_destroy(pool->c_handle); +out_c: kfree(pool); out: return NULL; @@ -626,6 +729,7 @@ static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, */ static void z3fold_destroy_pool(struct z3fold_pool *pool) { + kmem_cache_destroy(pool->c_handle); destroy_workqueue(pool->release_wq); destroy_workqueue(pool->compact_wq); kfree(pool); @@ -818,6 +922,7 @@ static void z3fold_free(struct z3fold_pool *pool, unsigned long handle) return; } + free_handle(handle); if (kref_put(&zhdr->refcount, release_z3fold_page_locked_list)) { atomic64_dec(&pool->pages_nr); return; From patchwork Wed Apr 17 08:39:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Wool X-Patchwork-Id: 10904977 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 338D713B5 for ; Wed, 17 Apr 2019 08:39:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16FDF28733 for ; Wed, 17 Apr 2019 08:39:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0B13E28A95; Wed, 17 Apr 2019 08:39:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F294828733 for ; Wed, 17 Apr 2019 08:39:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DC946B0008; Wed, 17 Apr 2019 04:39:28 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 189906B000A; Wed, 17 Apr 2019 04:39:28 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 031646B000D; Wed, 17 Apr 2019 04:39:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-lf1-f71.google.com (mail-lf1-f71.google.com [209.85.167.71]) by kanga.kvack.org (Postfix) with ESMTP id 85AA36B0008 for ; Wed, 17 Apr 2019 04:39:27 -0400 (EDT) Received: by mail-lf1-f71.google.com with SMTP id p13so3368701lfc.4 for ; Wed, 17 Apr 2019 01:39:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:from:to:cc:subject :message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=cSOQ7Gs++diSHQm1M9fFoOA1/dE9XvDkWRHKalGstic=; b=ZeAwlMQhOJ1qJFCB/4f2WjHk/sIE705jItL42oB0b1RLEOmEFohPQtRPkKt/7XKC20 uZCgDUBVkybL+PltLtu14QVVctEx1Uc6G1nmJFtQxxNL93GmVOk50WIZtCd5aTSqfd4k ZYDnbY0xixQgPTbTuMs9SXTePz1XX7FIT9WFzUklnX5R+oR9oaMh7aLTwXZ4jd2FCvtO N+qRtI6dxanrYNw9XKNbRUqwqlLqxjklVSYFRUsEYRvnDZiEX0BNQAS2hOWVDDOwwdJk lFrBQ8frUvY9F2hli3C+909S1nL/ioAuNhFFh1zd8UMh3qdFnCB9F3k4joJKoGsJzVrk wN+g== X-Gm-Message-State: APjAAAWQYOCyT2IHYrZQwAnQA3d/QaMxV3q/6LHQKTpjP9nn6D96eCNy p3eSVXt0njeb2HwJ97jhFCuuRAvWb/3bn+mgTQpde0smZWbn4ECIEg37gXuUisSyCLCHzx0oSTt RwOCfL3I2BBU5XmNsNMLN8vEy2+72sz7EMUxLzRW9QQDw3CbRR2hwhp14szo1EliayA== X-Received: by 2002:a2e:8089:: with SMTP id i9mr46629944ljg.137.1555490366933; Wed, 17 Apr 2019 01:39:26 -0700 (PDT) X-Received: by 2002:a2e:8089:: with SMTP id i9mr46629873ljg.137.1555490365153; Wed, 17 Apr 2019 01:39:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1555490365; cv=none; d=google.com; s=arc-20160816; b=NQfmto7fIzIY91SzpX5sRYu+3MK6XpMo427T9e3gfgPV6MZcjJ4cssPIDYcSkZpMWz b4m2MKPtT5rbnoVTUl2wllCCR+8gGE2P5jOiVPaiikeYDNnnjJrr3FP0QrEnf9KkZLps cBf2obx1SJDNsSYjm59RxES5CnAucfP3gSKEr+04VJDbx9EEBPR3zi2gv6pJJ4KBLSuB P0GoFLWGJAuA/jemyDMWdy/S7IYsrCR/aKcc5TMIWYP+boxp8092VZ6rnjsvXOPUOgYl sbWM04lUOHvVy2XweXCYOXylx6YDI0BLBiMDffteyp4hbRdNgrm/bIa5t3QyoV4UeR4I DgsA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:subject:cc:to:from:date:dkim-signature; bh=cSOQ7Gs++diSHQm1M9fFoOA1/dE9XvDkWRHKalGstic=; b=s+rsV2MHu0JVEccVj+S084kit0az8VpSin+l3f8Smr1SmoDRNdRnbeuIkmQ7JVCc7p XYmOGzuUjqm5XTiJaGeTWRB1c0DFsE3slBRSPmpR39Xc7Dhgz2FIKXuOD28WV2+vJA7p 0Dak+TZMnLNgLjIc+Aa7XvPGI6ap8la2UAlkdpg0YVlH3kAKs8FhHEWAaEsfePe/vVJW ElHucCKN7znV61ixKUkAH/BW37Ofr5+AdfmggJMmzQUGCCci5ym9mDfDXou2vNrVZiG0 xpLgPFq4jWGsMhJHLtp5dGQHywe+/CSShih+ljnCSEx5aVZIrEuSaOkzIzRCfCZINtFk e2oQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=oTc9RzZo; spf=pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=vitalywool@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id x26sor33039240ljb.16.2019.04.17.01.39.24 for (Google Transport Security); Wed, 17 Apr 2019 01:39:25 -0700 (PDT) Received-SPF: pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=oTc9RzZo; spf=pass (google.com: domain of vitalywool@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=vitalywool@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cSOQ7Gs++diSHQm1M9fFoOA1/dE9XvDkWRHKalGstic=; b=oTc9RzZo7AUyk5pj0tyHVttfdubvOnhHrx8eT0xZ8PbaQ+9PQTmqxjnWl3gpCFV7Xt Q9nvfRZF638cx7LG8/fw2Cu+kvndUtty56kC6eQreNhBp35P1pyZrgQipcLegyg+0lt+ tfjjOVmJGuw5WSwVnpV3vbFVuJF8P5oPV3fJBpolwtd2ta08rxW1AXoACAHV6A8pWdrx WSrB1Kr5rBx4J8VDZKJs6CW+suqL/dbD0x5uOklokgLAkv+u4PPU7I/o2JALZ7JcJhZv N3vpwdgSu22zSApgonIrJpjJuaxd75k8xc3HVAMtZVodC/luQ1FmSazWENtVgAoMeLpZ RZFA== X-Google-Smtp-Source: APXvYqwYJI2wYRIH1GvgDtpYP4W7t6wF+RkdlbsImDBBofg3hY0tgObM1O0O0TVDefCRoecArie5LA== X-Received: by 2002:a2e:8e96:: with SMTP id z22mr46203924ljk.123.1555490363999; Wed, 17 Apr 2019 01:39:23 -0700 (PDT) Received: from seldlx21914.corpusers.net ([37.139.156.40]) by smtp.gmail.com with ESMTPSA id t23sm10820473ljc.13.2019.04.17.01.39.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 17 Apr 2019 01:39:23 -0700 (PDT) Date: Wed, 17 Apr 2019 10:39:22 +0200 From: Vitaly Wool To: Linux-MM , linux-kernel@vger.kernel.org Cc: Dan Streetman , Andrew Morton , Oleksiy.Avramchenko@sony.com, Bartlomiej Zolnierkiewicz , Krzysztof Kozlowski Subject: [PATCHv2 4/4] z3fold: support page migration Message-Id: <20190417103922.31253da5c366c4ebe0419cfc@gmail.com> In-Reply-To: <20190417103510.36b055f3314e0e32b916b30a@gmail.com> References: <20190417103510.36b055f3314e0e32b916b30a@gmail.com> X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.30; x86_64-unknown-linux-gnu) Mime-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Now that we are not using page address in handles directly, we can make z3fold pages movable to decrease the memory fragmentation z3fold may create over time. This patch starts advertising non-headless z3fold pages as movable and uses the existing kernel infrastructure to implement moving of such pages per memory management subsystem's request. It thus implements 3 required callbacks for page migration: * isolation callback: z3fold_page_isolate(): try to isolate the page by removing it from all lists. Pages scheduled for some activity and mapped pages will not be isolated. Return true if isolation was successful or false otherwise * migration callback: z3fold_page_migrate(): re-check critical conditions and migrate page contents to the new page provided by the memory subsystem. Returns 0 on success or negative error code otherwise * putback callback: z3fold_page_putback(): put back the page if z3fold_page_migrate() for it failed permanently (i. e. not with -EAGAIN code). Signed-off-by: Vitaly Wool --- mm/z3fold.c | 241 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 231 insertions(+), 10 deletions(-) diff --git a/mm/z3fold.c b/mm/z3fold.c index bebc10083f1c..d9eabfdad0fe 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -24,10 +24,18 @@ #include #include +#include +#include #include #include #include +#include +#include +#include +#include #include +#include +#include #include #include #include @@ -97,6 +105,7 @@ struct z3fold_buddy_slots { * @middle_chunks: the size of the middle buddy in chunks, 0 if free * @last_chunks: the size of the last buddy in chunks, 0 if free * @first_num: the starting number (for the first handle) + * @mapped_count: the number of objects currently mapped */ struct z3fold_header { struct list_head buddy; @@ -110,6 +119,7 @@ struct z3fold_header { unsigned short last_chunks; unsigned short start_middle; unsigned short first_num:2; + unsigned short mapped_count:2; }; /** @@ -130,6 +140,7 @@ struct z3fold_header { * @compact_wq: workqueue for page layout background optimization * @release_wq: workqueue for safe page release * @work: work_struct for safe page release + * @inode: inode for z3fold pseudo filesystem * * This structure is allocated at pool creation time and maintains metadata * pertaining to a particular z3fold pool. @@ -149,6 +160,7 @@ struct z3fold_pool { struct workqueue_struct *compact_wq; struct workqueue_struct *release_wq; struct work_struct work; + struct inode *inode; }; /* @@ -227,6 +239,59 @@ static inline void free_handle(unsigned long handle) } } +static struct dentry *z3fold_do_mount(struct file_system_type *fs_type, + int flags, const char *dev_name, void *data) +{ + static const struct dentry_operations ops = { + .d_dname = simple_dname, + }; + + return mount_pseudo(fs_type, "z3fold:", NULL, &ops, 0x33); +} + +static struct file_system_type z3fold_fs = { + .name = "z3fold", + .mount = z3fold_do_mount, + .kill_sb = kill_anon_super, +}; + +static struct vfsmount *z3fold_mnt; +static int z3fold_mount(void) +{ + int ret = 0; + + z3fold_mnt = kern_mount(&z3fold_fs); + if (IS_ERR(z3fold_mnt)) + ret = PTR_ERR(z3fold_mnt); + + return ret; +} + +static void z3fold_unmount(void) +{ + kern_unmount(z3fold_mnt); +} + +static const struct address_space_operations z3fold_aops; +static int z3fold_register_migration(struct z3fold_pool *pool) +{ + pool->inode = alloc_anon_inode(z3fold_mnt->mnt_sb); + if (IS_ERR(pool->inode)) { + pool->inode = NULL; + return 1; + } + + pool->inode->i_mapping->private_data = pool; + pool->inode->i_mapping->a_ops = &z3fold_aops; + return 0; +} + +static void z3fold_unregister_migration(struct z3fold_pool *pool) +{ + if (pool->inode) + iput(pool->inode); + } + /* Initializes the z3fold header of a newly allocated z3fold page */ static struct z3fold_header *init_z3fold_page(struct page *page, struct z3fold_pool *pool) @@ -259,8 +324,14 @@ static struct z3fold_header *init_z3fold_page(struct page *page, } /* Resets the struct page fields and frees the page */ -static void free_z3fold_page(struct page *page) +static void free_z3fold_page(struct page *page, bool headless) { + if (!headless) { + lock_page(page); + __ClearPageMovable(page); + unlock_page(page); + } + ClearPagePrivate(page); __free_page(page); } @@ -317,12 +388,12 @@ static unsigned long encode_handle(struct z3fold_header *zhdr, enum buddy bud) } /* Returns the z3fold page where a given handle is stored */ -static inline struct z3fold_header *handle_to_z3fold_header(unsigned long handle) +static inline struct z3fold_header *handle_to_z3fold_header(unsigned long h) { - unsigned long addr = handle; + unsigned long addr = h; if (!(addr & (1 << PAGE_HEADLESS))) - addr = *(unsigned long *)handle; + addr = *(unsigned long *)h; return (struct z3fold_header *)(addr & PAGE_MASK); } @@ -366,7 +437,7 @@ static void __release_z3fold_page(struct z3fold_header *zhdr, bool locked) clear_bit(NEEDS_COMPACTING, &page->private); spin_lock(&pool->lock); if (!list_empty(&page->lru)) - list_del(&page->lru); + list_del_init(&page->lru); spin_unlock(&pool->lock); if (locked) z3fold_page_unlock(zhdr); @@ -420,7 +491,7 @@ static void free_pages_work(struct work_struct *w) continue; spin_unlock(&pool->stale_lock); cancel_work_sync(&zhdr->work); - free_z3fold_page(page); + free_z3fold_page(page, false); cond_resched(); spin_lock(&pool->stale_lock); } @@ -486,6 +557,9 @@ static int z3fold_compact_page(struct z3fold_header *zhdr) if (test_bit(MIDDLE_CHUNK_MAPPED, &page->private)) return 0; /* can't move middle chunk, it's used */ + if (unlikely(PageIsolated(page))) + return 0; + if (zhdr->middle_chunks == 0) return 0; /* nothing to compact */ @@ -546,6 +620,12 @@ static void do_compact_page(struct z3fold_header *zhdr, bool locked) return; } + if (unlikely(PageIsolated(page) || + test_bit(PAGE_STALE, &page->private))) { + z3fold_page_unlock(zhdr); + return; + } + z3fold_compact_page(zhdr); add_to_unbuddied(pool, zhdr); z3fold_page_unlock(zhdr); @@ -705,10 +785,14 @@ static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, pool->release_wq = create_singlethread_workqueue(pool->name); if (!pool->release_wq) goto out_wq; + if (z3fold_register_migration(pool)) + goto out_rwq; INIT_WORK(&pool->work, free_pages_work); pool->ops = ops; return pool; +out_rwq: + destroy_workqueue(pool->release_wq); out_wq: destroy_workqueue(pool->compact_wq); out_unbuddied: @@ -730,6 +814,7 @@ static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, static void z3fold_destroy_pool(struct z3fold_pool *pool) { kmem_cache_destroy(pool->c_handle); + z3fold_unregister_migration(pool); destroy_workqueue(pool->release_wq); destroy_workqueue(pool->compact_wq); kfree(pool); @@ -837,6 +922,7 @@ static int z3fold_alloc(struct z3fold_pool *pool, size_t size, gfp_t gfp, set_bit(PAGE_HEADLESS, &page->private); goto headless; } + __SetPageMovable(page, pool->inode->i_mapping); z3fold_page_lock(zhdr); found: @@ -895,7 +981,7 @@ static void z3fold_free(struct z3fold_pool *pool, unsigned long handle) spin_lock(&pool->lock); list_del(&page->lru); spin_unlock(&pool->lock); - free_z3fold_page(page); + free_z3fold_page(page, true); atomic64_dec(&pool->pages_nr); } return; @@ -931,7 +1017,8 @@ static void z3fold_free(struct z3fold_pool *pool, unsigned long handle) z3fold_page_unlock(zhdr); return; } - if (test_and_set_bit(NEEDS_COMPACTING, &page->private)) { + if (unlikely(PageIsolated(page)) || + test_and_set_bit(NEEDS_COMPACTING, &page->private)) { z3fold_page_unlock(zhdr); return; } @@ -1012,10 +1099,12 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) if (test_and_set_bit(PAGE_CLAIMED, &page->private)) continue; - zhdr = page_address(page); + if (unlikely(PageIsolated(page))) + continue; if (test_bit(PAGE_HEADLESS, &page->private)) break; + zhdr = page_address(page); if (!z3fold_page_trylock(zhdr)) { zhdr = NULL; continue; /* can't evict at this point */ @@ -1076,7 +1165,7 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) next: if (test_bit(PAGE_HEADLESS, &page->private)) { if (ret == 0) { - free_z3fold_page(page); + free_z3fold_page(page, true); atomic64_dec(&pool->pages_nr); return 0; } @@ -1153,6 +1242,8 @@ static void *z3fold_map(struct z3fold_pool *pool, unsigned long handle) break; } + if (addr) + zhdr->mapped_count++; z3fold_page_unlock(zhdr); out: return addr; @@ -1179,6 +1270,7 @@ static void z3fold_unmap(struct z3fold_pool *pool, unsigned long handle) buddy = handle_to_buddy(handle); if (buddy == MIDDLE) clear_bit(MIDDLE_CHUNK_MAPPED, &page->private); + zhdr->mapped_count--; z3fold_page_unlock(zhdr); } @@ -1193,6 +1285,128 @@ static u64 z3fold_get_pool_size(struct z3fold_pool *pool) return atomic64_read(&pool->pages_nr); } +bool z3fold_page_isolate(struct page *page, isolate_mode_t mode) +{ + struct z3fold_header *zhdr; + struct z3fold_pool *pool; + + VM_BUG_ON_PAGE(!PageMovable(page), page); + VM_BUG_ON_PAGE(PageIsolated(page), page); + + if (test_bit(PAGE_HEADLESS, &page->private)) + return false; + + zhdr = page_address(page); + z3fold_page_lock(zhdr); + if (test_bit(NEEDS_COMPACTING, &page->private) || + test_bit(PAGE_STALE, &page->private)) + goto out; + + pool = zhdr_to_pool(zhdr); + + if (zhdr->mapped_count == 0) { + kref_get(&zhdr->refcount); + if (!list_empty(&zhdr->buddy)) + list_del_init(&zhdr->buddy); + spin_lock(&pool->lock); + if (!list_empty(&page->lru)) + list_del(&page->lru); + spin_unlock(&pool->lock); + z3fold_page_unlock(zhdr); + return true; + } +out: + z3fold_page_unlock(zhdr); + return false; +} + +int z3fold_page_migrate(struct address_space *mapping, struct page *newpage, + struct page *page, enum migrate_mode mode) +{ + struct z3fold_header *zhdr, *new_zhdr; + struct z3fold_pool *pool; + struct address_space *new_mapping; + + VM_BUG_ON_PAGE(!PageMovable(page), page); + VM_BUG_ON_PAGE(!PageIsolated(page), page); + + zhdr = page_address(page); + pool = zhdr_to_pool(zhdr); + + if (!trylock_page(page)) + return -EAGAIN; + + if (!z3fold_page_trylock(zhdr)) { + unlock_page(page); + return -EAGAIN; + } + if (zhdr->mapped_count != 0) { + z3fold_page_unlock(zhdr); + unlock_page(page); + return -EBUSY; + } + new_zhdr = page_address(newpage); + memcpy(new_zhdr, zhdr, PAGE_SIZE); + newpage->private = page->private; + page->private = 0; + z3fold_page_unlock(zhdr); + spin_lock_init(&new_zhdr->page_lock); + new_mapping = page_mapping(page); + __ClearPageMovable(page); + ClearPagePrivate(page); + + get_page(newpage); + z3fold_page_lock(new_zhdr); + if (new_zhdr->first_chunks) + encode_handle(new_zhdr, FIRST); + if (new_zhdr->last_chunks) + encode_handle(new_zhdr, LAST); + if (new_zhdr->middle_chunks) + encode_handle(new_zhdr, MIDDLE); + set_bit(NEEDS_COMPACTING, &newpage->private); + new_zhdr->cpu = smp_processor_id(); + spin_lock(&pool->lock); + list_add(&newpage->lru, &pool->lru); + spin_unlock(&pool->lock); + __SetPageMovable(newpage, new_mapping); + z3fold_page_unlock(new_zhdr); + + queue_work_on(new_zhdr->cpu, pool->compact_wq, &new_zhdr->work); + + page_mapcount_reset(page); + unlock_page(page); + put_page(page); + return 0; +} + +void z3fold_page_putback(struct page *page) +{ + struct z3fold_header *zhdr; + struct z3fold_pool *pool; + + zhdr = page_address(page); + pool = zhdr_to_pool(zhdr); + + z3fold_page_lock(zhdr); + if (!list_empty(&zhdr->buddy)) + list_del_init(&zhdr->buddy); + INIT_LIST_HEAD(&page->lru); + if (kref_put(&zhdr->refcount, release_z3fold_page_locked)) { + atomic64_dec(&pool->pages_nr); + return; + } + spin_lock(&pool->lock); + list_add(&page->lru, &pool->lru); + spin_unlock(&pool->lock); + z3fold_page_unlock(zhdr); +} + +static const struct address_space_operations z3fold_aops = { + .isolate_page = z3fold_page_isolate, + .migratepage = z3fold_page_migrate, + .putback_page = z3fold_page_putback, +}; + /***************** * zpool ****************/ @@ -1290,8 +1504,14 @@ MODULE_ALIAS("zpool-z3fold"); static int __init init_z3fold(void) { + int ret; + /* Make sure the z3fold header is not larger than the page size */ BUILD_BUG_ON(ZHDR_SIZE_ALIGNED > PAGE_SIZE); + ret = z3fold_mount(); + if (ret) + return ret; + zpool_register_driver(&z3fold_zpool_driver); return 0; @@ -1299,6 +1519,7 @@ static int __init init_z3fold(void) static void __exit exit_z3fold(void) { + z3fold_unmount(); zpool_unregister_driver(&z3fold_zpool_driver); }