From patchwork Fri Jun 7 06:07:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 10980803 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 062DB6C5 for ; Fri, 7 Jun 2019 06:08:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E933328B04 for ; Fri, 7 Jun 2019 06:08:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DD0A928B0F; Fri, 7 Jun 2019 06:08:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1723028B04 for ; Fri, 7 Jun 2019 06:08:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7A11B6B0271; Fri, 7 Jun 2019 02:08:27 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 703BD6B0272; Fri, 7 Jun 2019 02:08:27 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 507E26B0273; Fri, 7 Jun 2019 02:08:27 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f199.google.com (mail-pg1-f199.google.com [209.85.215.199]) by kanga.kvack.org (Postfix) with ESMTP id 071256B0272 for ; Fri, 7 Jun 2019 02:08:27 -0400 (EDT) Received: by mail-pg1-f199.google.com with SMTP id g9so720552pgd.17 for ; Thu, 06 Jun 2019 23:08:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=lrPPZ8NiUUMLO+yAkhLtS4Iy/Ao4Mc5vCsF4KS5LBdo=; b=GmsPyUZUmxtSSPpn07sjYxWCj0ysIiLqQ+nzeu7+t7hJ3dO0xQYNlm24Cih4p/E9EX WReSltq209K5WK8wJHv+y+mOvceInh4DVH28WqTC9WDEEAe+fx0mAishzS+/fRwZ3Bpv ANybiyfmBbPeUdLaj4edXwq47dmoICUEXaNOCQDZhXEnQnHCYbZiUg99s+yyTKE0MbM+ ud+JO4t8yt684HH4sxyIcweVdbvEhptJJhPrqtos7vZXc4fE+IXWHCzMB9L6kjM6tf8x 5lvSpxeaJim0aNzEFqwQ64UOV2WZTm0JJ7U5hLOM6Ugyv52n+obT+Fu1ksle2OoiQXDY mn5A== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: APjAAAWxtcqMNjPgoR9txIAO1zwF1W+7sWJ3pOqRluOIY7P4Yrzsbs7J lnPOeTK15CEL848D+KZUz+tmh0uzpi3iJGXjzbl7OsPl68brg8bgz1wkHNmSX1DBe62o5fgthu2 7uVIKFQG1ry0FSvt5q6kX0L5WxTxdOlTdvT3Ir5fWtWTWBmfLnKdzauZK7WTJeAjHrw== X-Received: by 2002:a17:90a:9503:: with SMTP id t3mr3687944pjo.47.1559887706476; Thu, 06 Jun 2019 23:08:26 -0700 (PDT) X-Google-Smtp-Source: APXvYqyVOH5UALYMJZeL+FfL0bYWNAOPl09kno2cfi2gTjYjsIp3l49ulvYfDlGPG18339dRu8px X-Received: by 2002:a17:90a:9503:: with SMTP id t3mr3687876pjo.47.1559887705243; Thu, 06 Jun 2019 23:08:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1559887705; cv=none; d=google.com; s=arc-20160816; b=OwPF5QIGAsGs6nyfn/WItj8vUOkrX/vLfTPaYmU7/yucsYzG8VaQjuk+YveRBqstme hCsJ3koxIDGhoMX/J2QIjbjPgqbPaETk/S4/dP0kGOybFSRLMC0x3+T7n6k93RuIYIpF jHeUaNxaGZxstgSR9riQ4tSsbpao3/espT6kp5rraXLjaVX7AWL+agIMoPBSEfWf2hrh bD6nGKInz7JIM8z+CvZsX3N2g3JobXOCzS8BZ678SpkJUWgRBF3o7nnw0WTrui25gUq4 0ASji/mtbwoEgQ2upU3RrIMMyE4yXd7wudCT85K3VPzvYiOJ/5o5as0HzW6j3z02h8L4 aRiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=lrPPZ8NiUUMLO+yAkhLtS4Iy/Ao4Mc5vCsF4KS5LBdo=; b=G90nTwCAioNPGmFmmYcDogGwlTHpUf87PWKYaUKYmkZJfLL1MvG4T4HDCEbXehGB72 DsnSOPRY9BajWUPUtU4QwGOnRWEtQbS1u/0+iybkhMfylE2ZofXJf8huklKi3MGH/nRF a1Kvp+aUs6F6uvbtA/FsDfawJC/vOStOIpwDHK7q94qB9XpDvpoOOekVeZV/b2KFAzPK PcmapmEZmVkaOS2TiTRF8DAG81wmW/6o0f3l3CeV8ksbnc0SYg0ZwqahUiTMQogKRxcT JVmlrrfWsD9gbwltg68/EneF7HzCcyaHRnZHC53htkdPcq58U63fTJULIwWP7aN/AZF7 amGw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out4436.biz.mail.alibaba.com (out4436.biz.mail.alibaba.com. [47.88.44.36]) by mx.google.com with ESMTPS id q189si1013925pga.156.2019.06.06.23.08.23 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 06 Jun 2019 23:08:25 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) client-ip=47.88.44.36; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01f04446;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0TTcZLUN_1559887677; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0TTcZLUN_1559887677) by smtp.aliyun-inc.com(127.0.0.1); Fri, 07 Jun 2019 14:08:10 +0800 From: Yang Shi To: ktkhai@virtuozzo.com, kirill.shutemov@linux.intel.com, hannes@cmpxchg.org, mhocko@suse.com, hughd@google.com, shakeelb@google.com, rientjes@google.com, akpm@linux-foundation.org Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/4] mm: thp: extract split_queue_* into a struct Date: Fri, 7 Jun 2019 14:07:36 +0800 Message-Id: <1559887659-23121-2-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1559887659-23121-1-git-send-email-yang.shi@linux.alibaba.com> References: <1559887659-23121-1-git-send-email-yang.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Put split_queue, split_queue_lock and split_queue_len into a struct in order to reduce code duplication when we convert deferred_split to memcg aware in the later patches. Suggested-by: "Kirill A . Shutemov" Cc: Kirill Tkhai Cc: Johannes Weiner Cc: Michal Hocko Cc: Hugh Dickins Cc: Shakeel Butt Cc: David Rientjes Signed-off-by: Yang Shi --- include/linux/mmzone.h | 12 +++++++++--- mm/huge_memory.c | 45 +++++++++++++++++++++++++-------------------- mm/page_alloc.c | 8 +++++--- 3 files changed, 39 insertions(+), 26 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 70394ca..7799166 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -676,6 +676,14 @@ struct zonelist { extern struct page *mem_map; #endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +struct deferred_split { + spinlock_t split_queue_lock; + struct list_head split_queue; + unsigned long split_queue_len; +}; +#endif + /* * On NUMA machines, each NUMA node would have a pg_data_t to describe * it's memory layout. On UMA machines there is a single pglist_data which @@ -755,9 +763,7 @@ struct zonelist { #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE - spinlock_t split_queue_lock; - struct list_head split_queue; - unsigned long split_queue_len; + struct deferred_split deferred_split_queue; #endif /* Fields commonly accessed by the page reclaim scanner */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9f8bce9..81cf759 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2658,6 +2658,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) { struct page *head = compound_head(page); struct pglist_data *pgdata = NODE_DATA(page_to_nid(head)); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int count, mapcount, extra_pins, ret; @@ -2744,17 +2745,17 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) } /* Prevent deferred_split_scan() touching ->_refcount */ - spin_lock(&pgdata->split_queue_lock); + spin_lock(&ds_queue->split_queue_lock); count = page_count(head); mapcount = total_mapcount(head); if (!mapcount && page_ref_freeze(head, 1 + extra_pins)) { if (!list_empty(page_deferred_list(head))) { - pgdata->split_queue_len--; + ds_queue->split_queue_len--; list_del(page_deferred_list(head)); } if (mapping) __dec_node_page_state(page, NR_SHMEM_THPS); - spin_unlock(&pgdata->split_queue_lock); + spin_unlock(&ds_queue->split_queue_lock); __split_huge_page(page, list, end, flags); if (PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; @@ -2771,7 +2772,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) dump_page(page, "total_mapcount(head) > 0"); BUG(); } - spin_unlock(&pgdata->split_queue_lock); + spin_unlock(&ds_queue->split_queue_lock); fail: if (mapping) xa_unlock(&mapping->i_pages); spin_unlock_irqrestore(&pgdata->lru_lock, flags); @@ -2794,52 +2795,56 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) void free_transhuge_page(struct page *page) { struct pglist_data *pgdata = NODE_DATA(page_to_nid(page)); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; unsigned long flags; - spin_lock_irqsave(&pgdata->split_queue_lock, flags); + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); if (!list_empty(page_deferred_list(page))) { - pgdata->split_queue_len--; + ds_queue->split_queue_len--; list_del(page_deferred_list(page)); } - spin_unlock_irqrestore(&pgdata->split_queue_lock, flags); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); free_compound_page(page); } void deferred_split_huge_page(struct page *page) { struct pglist_data *pgdata = NODE_DATA(page_to_nid(page)); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; unsigned long flags; VM_BUG_ON_PAGE(!PageTransHuge(page), page); - spin_lock_irqsave(&pgdata->split_queue_lock, flags); + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); if (list_empty(page_deferred_list(page))) { count_vm_event(THP_DEFERRED_SPLIT_PAGE); - list_add_tail(page_deferred_list(page), &pgdata->split_queue); - pgdata->split_queue_len++; + list_add_tail(page_deferred_list(page), &ds_queue->split_queue); + ds_queue->split_queue_len++; } - spin_unlock_irqrestore(&pgdata->split_queue_lock, flags); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } static unsigned long deferred_split_count(struct shrinker *shrink, struct shrink_control *sc) { struct pglist_data *pgdata = NODE_DATA(sc->nid); - return READ_ONCE(pgdata->split_queue_len); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; + return READ_ONCE(ds_queue->split_queue_len); } static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc) { struct pglist_data *pgdata = NODE_DATA(sc->nid); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; unsigned long flags; LIST_HEAD(list), *pos, *next; struct page *page; int split = 0; - spin_lock_irqsave(&pgdata->split_queue_lock, flags); + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); /* Take pin on all head pages to avoid freeing them under us */ - list_for_each_safe(pos, next, &pgdata->split_queue) { + list_for_each_safe(pos, next, &ds_queue->split_queue) { page = list_entry((void *)pos, struct page, mapping); page = compound_head(page); if (get_page_unless_zero(page)) { @@ -2847,12 +2852,12 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, } else { /* We lost race with put_compound_page() */ list_del_init(page_deferred_list(page)); - pgdata->split_queue_len--; + ds_queue->split_queue_len--; } if (!--sc->nr_to_scan) break; } - spin_unlock_irqrestore(&pgdata->split_queue_lock, flags); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); list_for_each_safe(pos, next, &list) { page = list_entry((void *)pos, struct page, mapping); @@ -2866,15 +2871,15 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, put_page(page); } - spin_lock_irqsave(&pgdata->split_queue_lock, flags); - list_splice_tail(&list, &pgdata->split_queue); - spin_unlock_irqrestore(&pgdata->split_queue_lock, flags); + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + list_splice_tail(&list, &ds_queue->split_queue); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); /* * Stop shrinker if we didn't split any page, but the queue is empty. * This can happen if pages were freed under us. */ - if (!split && list_empty(&pgdata->split_queue)) + if (!split && list_empty(&ds_queue->split_queue)) return SHRINK_STOP; return split; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3b13d39..a82104a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6581,9 +6581,11 @@ static unsigned long __init calc_memmap_size(unsigned long spanned_pages, #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pgdat_init_split_queue(struct pglist_data *pgdat) { - spin_lock_init(&pgdat->split_queue_lock); - INIT_LIST_HEAD(&pgdat->split_queue); - pgdat->split_queue_len = 0; + struct deferred_split *ds_queue = &pgdat->deferred_split_queue; + + spin_lock_init(&ds_queue->split_queue_lock); + INIT_LIST_HEAD(&ds_queue->split_queue); + ds_queue->split_queue_len = 0; } #else static void pgdat_init_split_queue(struct pglist_data *pgdat) {}