From patchwork Wed Jun 26 00:02:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11016615 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8F56E14C0 for ; Wed, 26 Jun 2019 00:03:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D90F284DC for ; Wed, 26 Jun 2019 00:03:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 711C5285D9; Wed, 26 Jun 2019 00:03:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A3146284DC for ; Wed, 26 Jun 2019 00:03:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CECE06B0007; Tue, 25 Jun 2019 20:03:09 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id CCD128E0002; Tue, 25 Jun 2019 20:03:09 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8B038E0002; Tue, 25 Jun 2019 20:03:09 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) by kanga.kvack.org (Postfix) with ESMTP id 803586B0007 for ; Tue, 25 Jun 2019 20:03:09 -0400 (EDT) Received: by mail-pg1-f197.google.com with SMTP id o19so355929pgl.14 for ; Tue, 25 Jun 2019 17:03:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=lc9EZ7km2nMW/LkyUnIW6C5nTmcnU2WKuYzRTkNIjZo=; b=pb2GUxrkR+iz95XwKrmtoQAaibhhHeVaVqklt2Xr1LZhMQoWAdT+0KrNoMl4jTkn0Y VEFS1QxM6OQOn2kGeQvmqsvUjDs0pv7wOvXGO0OmgvzF1BWpLHd5bjrVPttFVGTrds/K x9+JBVy1y3rX1LpLrCXFzbcT9xX1N+TlkRbH8lIDgM7Te+TbSqv7WS6EAPHSfMAb/8mv hw9yJxIeCSfsVF7zsCmP+Q0a4DerlnIi74s0ZlkImit8xShJALziEcXHxVhY/e0zQ8yx o+kAF7/jsjzSzCtujCerTlOOskua0OWj9pRZhesQTFJtKRJzUVZVaka5bxOOh5lr3OnS q7Qg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: APjAAAW3BmdoSbExtQxLBqodPUWJMUtJD4bn1lCFznYIqcly6vX4VeV6 k1nJbMeVQ9pcSQj7BGJN2vb6gZat3yF6SZf79wz5YOdpIVFv4v+qQ3zW0PvkhlHeuEYUgXqkOmD dIVIRzYwfTCeFYBa+PV54ns8YB3yqm8r99fsdCZsg7ZlxOtsxNNIHNb4XD2kNMVbHiA== X-Received: by 2002:a63:3d02:: with SMTP id k2mr4191792pga.36.1561507388996; Tue, 25 Jun 2019 17:03:08 -0700 (PDT) X-Google-Smtp-Source: APXvYqzH0n9uNsjFitSnIP6DLPPZZ6ehVnoK8b9H/zRoAj+YQygMBhbeeopikQs26AjE8CJ9QqYd X-Received: by 2002:a63:3d02:: with SMTP id k2mr4191687pga.36.1561507387622; Tue, 25 Jun 2019 17:03:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561507387; cv=none; d=google.com; s=arc-20160816; b=WYNe1z4f5vAw0z36wB3V5I2sc2Gisc/y3+Czo+UhDrWrzmtbAystcZL839TdjIpN0g JsTkZbEUwO+Mni4bSlrBFJdVTaL63po+ypOdYnxq4iElngy38d5yjazR6GHy5PxzJlAI L46Bv/Rb04NpyA0J7+SDRtzb/st126o0QXIci40K9bZULeGT8KtKY0n2YfhpcH5p/NX5 65GpYJx0Q14lnMZbbCCed4U+i1fB+5R4D/dG9L75YDq01J6v3r+jr1LgZV4PHfd16K0z 5pCBAcPae/wDltoUQTSHZRocxSspshIJH77g69BUpScWILYmYFn26HGKic0DYGFTHej/ KNdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=lc9EZ7km2nMW/LkyUnIW6C5nTmcnU2WKuYzRTkNIjZo=; b=L1RNR3Z/aotCorajISvq47wfjo0EMzXJTVa9MUIEqjh8Kv89vJzYPQhCMTdy8lPo3w Kz0GQIAJVGuSdt694qUGVse70L/XnXIIicHt40TLsHVxTqCc2w6Dc0qJvrbjVVFvfaps u1W1CfYCCldfFtzoE9E+RIqZnyd1xMdBP+HEvTTXZqVwr8pyigIeg6TFxnC7ty2TR3jt +OQ4ZU4xbTh3bkj5Oe5/cR7DcYJuErWEi+x9F2SzZ2hd7YjYF40PxR88gC1U4o53KZjt rbQo/ALdXWQ69U8TBkpVpr1JcXCuhvUKDQM1wP3VWQ5RW/sqOFPN9fd2cM1qls6RArX8 3Tpg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out4436.biz.mail.alibaba.com (out4436.biz.mail.alibaba.com. [47.88.44.36]) by mx.google.com with ESMTPS id k22si14611865pfk.90.2019.06.25.17.03.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Jun 2019 17:03:07 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) client-ip=47.88.44.36; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.36 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R761e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0TVCYVJX_1561507375; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0TVCYVJX_1561507375) by smtp.aliyun-inc.com(127.0.0.1); Wed, 26 Jun 2019 08:03:02 +0800 From: Yang Shi To: kirill.shutemov@linux.intel.com, ktkhai@virtuozzo.com, hannes@cmpxchg.org, mhocko@suse.com, hughd@google.com, shakeelb@google.com, rientjes@google.com, akpm@linux-foundation.org Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 1/4] mm: thp: extract split_queue_* into a struct Date: Wed, 26 Jun 2019 08:02:38 +0800 Message-Id: <1561507361-59349-2-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1561507361-59349-1-git-send-email-yang.shi@linux.alibaba.com> References: <1561507361-59349-1-git-send-email-yang.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Put split_queue, split_queue_lock and split_queue_len into a struct in order to reduce code duplication when we convert deferred_split to memcg aware in the later patches. Suggested-by: "Kirill A . Shutemov" Cc: Kirill Tkhai Cc: Johannes Weiner Cc: Michal Hocko Cc: Hugh Dickins Cc: Shakeel Butt Cc: David Rientjes Acked-by: Kirill A. Shutemov Signed-off-by: Yang Shi --- include/linux/mmzone.h | 12 +++++++++--- mm/huge_memory.c | 45 +++++++++++++++++++++++++-------------------- mm/page_alloc.c | 8 +++++--- 3 files changed, 39 insertions(+), 26 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 70394ca..7799166 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -676,6 +676,14 @@ struct zonelist { extern struct page *mem_map; #endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +struct deferred_split { + spinlock_t split_queue_lock; + struct list_head split_queue; + unsigned long split_queue_len; +}; +#endif + /* * On NUMA machines, each NUMA node would have a pg_data_t to describe * it's memory layout. On UMA machines there is a single pglist_data which @@ -755,9 +763,7 @@ struct zonelist { #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE - spinlock_t split_queue_lock; - struct list_head split_queue; - unsigned long split_queue_len; + struct deferred_split deferred_split_queue; #endif /* Fields commonly accessed by the page reclaim scanner */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9f8bce9..81cf759 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2658,6 +2658,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) { struct page *head = compound_head(page); struct pglist_data *pgdata = NODE_DATA(page_to_nid(head)); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int count, mapcount, extra_pins, ret; @@ -2744,17 +2745,17 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) } /* Prevent deferred_split_scan() touching ->_refcount */ - spin_lock(&pgdata->split_queue_lock); + spin_lock(&ds_queue->split_queue_lock); count = page_count(head); mapcount = total_mapcount(head); if (!mapcount && page_ref_freeze(head, 1 + extra_pins)) { if (!list_empty(page_deferred_list(head))) { - pgdata->split_queue_len--; + ds_queue->split_queue_len--; list_del(page_deferred_list(head)); } if (mapping) __dec_node_page_state(page, NR_SHMEM_THPS); - spin_unlock(&pgdata->split_queue_lock); + spin_unlock(&ds_queue->split_queue_lock); __split_huge_page(page, list, end, flags); if (PageSwapCache(head)) { swp_entry_t entry = { .val = page_private(head) }; @@ -2771,7 +2772,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) dump_page(page, "total_mapcount(head) > 0"); BUG(); } - spin_unlock(&pgdata->split_queue_lock); + spin_unlock(&ds_queue->split_queue_lock); fail: if (mapping) xa_unlock(&mapping->i_pages); spin_unlock_irqrestore(&pgdata->lru_lock, flags); @@ -2794,52 +2795,56 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) void free_transhuge_page(struct page *page) { struct pglist_data *pgdata = NODE_DATA(page_to_nid(page)); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; unsigned long flags; - spin_lock_irqsave(&pgdata->split_queue_lock, flags); + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); if (!list_empty(page_deferred_list(page))) { - pgdata->split_queue_len--; + ds_queue->split_queue_len--; list_del(page_deferred_list(page)); } - spin_unlock_irqrestore(&pgdata->split_queue_lock, flags); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); free_compound_page(page); } void deferred_split_huge_page(struct page *page) { struct pglist_data *pgdata = NODE_DATA(page_to_nid(page)); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; unsigned long flags; VM_BUG_ON_PAGE(!PageTransHuge(page), page); - spin_lock_irqsave(&pgdata->split_queue_lock, flags); + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); if (list_empty(page_deferred_list(page))) { count_vm_event(THP_DEFERRED_SPLIT_PAGE); - list_add_tail(page_deferred_list(page), &pgdata->split_queue); - pgdata->split_queue_len++; + list_add_tail(page_deferred_list(page), &ds_queue->split_queue); + ds_queue->split_queue_len++; } - spin_unlock_irqrestore(&pgdata->split_queue_lock, flags); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } static unsigned long deferred_split_count(struct shrinker *shrink, struct shrink_control *sc) { struct pglist_data *pgdata = NODE_DATA(sc->nid); - return READ_ONCE(pgdata->split_queue_len); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; + return READ_ONCE(ds_queue->split_queue_len); } static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc) { struct pglist_data *pgdata = NODE_DATA(sc->nid); + struct deferred_split *ds_queue = &pgdata->deferred_split_queue; unsigned long flags; LIST_HEAD(list), *pos, *next; struct page *page; int split = 0; - spin_lock_irqsave(&pgdata->split_queue_lock, flags); + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); /* Take pin on all head pages to avoid freeing them under us */ - list_for_each_safe(pos, next, &pgdata->split_queue) { + list_for_each_safe(pos, next, &ds_queue->split_queue) { page = list_entry((void *)pos, struct page, mapping); page = compound_head(page); if (get_page_unless_zero(page)) { @@ -2847,12 +2852,12 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, } else { /* We lost race with put_compound_page() */ list_del_init(page_deferred_list(page)); - pgdata->split_queue_len--; + ds_queue->split_queue_len--; } if (!--sc->nr_to_scan) break; } - spin_unlock_irqrestore(&pgdata->split_queue_lock, flags); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); list_for_each_safe(pos, next, &list) { page = list_entry((void *)pos, struct page, mapping); @@ -2866,15 +2871,15 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, put_page(page); } - spin_lock_irqsave(&pgdata->split_queue_lock, flags); - list_splice_tail(&list, &pgdata->split_queue); - spin_unlock_irqrestore(&pgdata->split_queue_lock, flags); + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + list_splice_tail(&list, &ds_queue->split_queue); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); /* * Stop shrinker if we didn't split any page, but the queue is empty. * This can happen if pages were freed under us. */ - if (!split && list_empty(&pgdata->split_queue)) + if (!split && list_empty(&ds_queue->split_queue)) return SHRINK_STOP; return split; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index d66bc8a..6c9cf1e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6582,9 +6582,11 @@ static unsigned long __init calc_memmap_size(unsigned long spanned_pages, #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void pgdat_init_split_queue(struct pglist_data *pgdat) { - spin_lock_init(&pgdat->split_queue_lock); - INIT_LIST_HEAD(&pgdat->split_queue); - pgdat->split_queue_len = 0; + struct deferred_split *ds_queue = &pgdat->deferred_split_queue; + + spin_lock_init(&ds_queue->split_queue_lock); + INIT_LIST_HEAD(&ds_queue->split_queue); + ds_queue->split_queue_len = 0; } #else static void pgdat_init_split_queue(struct pglist_data *pgdat) {} From patchwork Wed Jun 26 00:02:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11016613 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4329014C0 for ; Wed, 26 Jun 2019 00:03:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 331B8284DC for ; Wed, 26 Jun 2019 00:03:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 26B16285D9; Wed, 26 Jun 2019 00:03:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D101284DC for ; Wed, 26 Jun 2019 00:03:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FF556B0006; Tue, 25 Jun 2019 20:03:07 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2B1A98E0003; Tue, 25 Jun 2019 20:03:07 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19F2A8E0002; Tue, 25 Jun 2019 20:03:07 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by kanga.kvack.org (Postfix) with ESMTP id D78C46B0006 for ; Tue, 25 Jun 2019 20:03:06 -0400 (EDT) Received: by mail-pf1-f197.google.com with SMTP id 145so343140pfv.18 for ; Tue, 25 Jun 2019 17:03:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=6CrxZMPVKtWiO2W03kvINqv+b3cQRf9/AK6f9stpruQ=; b=UZJm2u17PomnP7dKoYQCngG5ZALvms6Wojkq/kF7P85UVMZ4C42+O8OUC52eIIbwVN x5UjEHZu11azqkDf4CJ2CWe/2EG9jmVEgeJ9EODoocOEy1sUpoTlanA9ma3F6VkAbEQO NwaM469HDitwWaSTVnqS/BDpcxwwpuMVDtHwofsrkbk1f4ZVhOC+RHBaUQDjLkgILL5g b/Gtu/Oz6ZhOTuPCMbKSvumgeVlF6zBWX/TlqjP66ErsTv1RlWGPf42C7hjOSCq6rfay MsLW9pRXhSXoKW7pjpfgKyBPH7tKkNddjy5vtQxxHApnG1HbPtI+oWCxaTGcP/2jiUee Z2nA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.44 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: APjAAAWvQiLGhO6b/j0hBr1F41myvoT4Lg6tq6RLVnzHE3CXtqKKN6bG uKXHrNMBa9zi9fPGTdJd9K6Sy2leHih2twjlpHbGkZYDOEhoYjkEs631VA6vpjGk6V8zKfvG/LG yrlEzwCKJ2MK8fch1isPZwHTQIDsZNS8VPW0yU1kp4Etq00b1dBG7yEo7iwAwZ9o5qA== X-Received: by 2002:a17:90a:bb94:: with SMTP id v20mr680125pjr.88.1561507386452; Tue, 25 Jun 2019 17:03:06 -0700 (PDT) X-Google-Smtp-Source: APXvYqzMC7hgLkq3aL01USZjA+AH723n4STuvM6QVq66jtAOER3heLIU0Lf+QlqhT6MpvyvM0AUP X-Received: by 2002:a17:90a:bb94:: with SMTP id v20mr680030pjr.88.1561507385419; Tue, 25 Jun 2019 17:03:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561507385; cv=none; d=google.com; s=arc-20160816; b=glKUNUIUj2oRhr7fq4z7Y9Swd33Wg83jcXllg9kMLdu6zpq0DYBE9c/RxNkFVZImAe ScWKKFjm5zS93W4VhYuvJXAVf08kJoZJlxPXsKNNCDf+TaY4ss1ASoqq5yXanw0buL/b TH18XxeyQpqm0FSzB8SySuz97EmWDJjypRUWfJ+Xq/aVDylb2Fveru9d2m0x2TuXlGKH zitaJcK1m0PXZkZ1+YWmkRgsY2pYrAD7IXC3AbXwXOSy8nCdjoNBU3mJ9F1Z7fYwva11 IMk2V+m/sxW9kwLEXFNF++zsiKL94JOfDt1qnVLHOHKFnESByLvQtzebzAfMsnB7GkTO 7Rag== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=6CrxZMPVKtWiO2W03kvINqv+b3cQRf9/AK6f9stpruQ=; b=rFER6JJ5lF/JUuNVvpEWqRSziQRF1kzXXOZE3ac9RgC7dtjuePF2MMe0s3wsp4gASZ PL2oIGIHh4TMHhT4qDIfZCIqTGe4bGvMy5QaHQotexgsI+U0i4OQeU76ouYY4U36/ccJ o2d1J4yboiOLYuP8qRzZxdTXa3PcLLAb9H2074rixD2Wm9wb/ggk/WK8Vk1bqjwWxvpQ 879oDJrvplaa7FyoNxrx4c2ZwsMemsUY9Q3ju/GnIkHG9ANc657AC6/ivwIqha3vWPJ3 59mlzH9RvkAVFY8tcMj7WQUawIrsONZYwg+tU+AHkhCrmocerie0RTX8DWtG7RAPQpay HjNQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.44 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com. [115.124.30.44]) by mx.google.com with ESMTPS id h12si12067947pfn.171.2019.06.25.17.03.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Jun 2019 17:03:05 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.44 as permitted sender) client-ip=115.124.30.44; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.44 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R411e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04394;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0TVCYVJX_1561507375; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0TVCYVJX_1561507375) by smtp.aliyun-inc.com(127.0.0.1); Wed, 26 Jun 2019 08:03:03 +0800 From: Yang Shi To: kirill.shutemov@linux.intel.com, ktkhai@virtuozzo.com, hannes@cmpxchg.org, mhocko@suse.com, hughd@google.com, shakeelb@google.com, rientjes@google.com, akpm@linux-foundation.org Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 2/4] mm: move mem_cgroup_uncharge out of __page_cache_release() Date: Wed, 26 Jun 2019 08:02:39 +0800 Message-Id: <1561507361-59349-3-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1561507361-59349-1-git-send-email-yang.shi@linux.alibaba.com> References: <1561507361-59349-1-git-send-email-yang.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The later patch would make THP deferred split shrinker memcg aware, but it needs page->mem_cgroup information in THP destructor, which is called after mem_cgroup_uncharge() now. So, move mem_cgroup_uncharge() from __page_cache_release() to compound page destructor, which is called by both THP and other compound pages except HugeTLB. And call it in __put_single_page() for single order page. Suggested-by: "Kirill A . Shutemov" Cc: Kirill Tkhai Cc: Johannes Weiner Cc: Michal Hocko Cc: Hugh Dickins Cc: Shakeel Butt Cc: David Rientjes Acked-by: Kirill A. Shutemov Signed-off-by: Yang Shi --- mm/page_alloc.c | 1 + mm/swap.c | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6c9cf1e..53a7a6c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -624,6 +624,7 @@ static void bad_page(struct page *page, const char *reason, void free_compound_page(struct page *page) { + mem_cgroup_uncharge(page); __free_pages_ok(page, compound_order(page)); } diff --git a/mm/swap.c b/mm/swap.c index 7ede3ed..170a725 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -71,12 +71,12 @@ static void __page_cache_release(struct page *page) spin_unlock_irqrestore(&pgdat->lru_lock, flags); } __ClearPageWaiters(page); - mem_cgroup_uncharge(page); } static void __put_single_page(struct page *page) { __page_cache_release(page); + mem_cgroup_uncharge(page); free_unref_page(page); } From patchwork Wed Jun 26 00:02:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11016643 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9339014BB for ; Wed, 26 Jun 2019 00:04:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E691285DD for ; Wed, 26 Jun 2019 00:04:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6C786285E0; Wed, 26 Jun 2019 00:04:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D0D49284DC for ; Wed, 26 Jun 2019 00:04:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12A636B0003; Tue, 25 Jun 2019 20:04:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0DC608E0003; Tue, 25 Jun 2019 20:04:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F33638E0002; Tue, 25 Jun 2019 20:04:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-ot1-f71.google.com (mail-ot1-f71.google.com [209.85.210.71]) by kanga.kvack.org (Postfix) with ESMTP id CEF186B0003 for ; Tue, 25 Jun 2019 20:04:10 -0400 (EDT) Received: by mail-ot1-f71.google.com with SMTP id m16so186127otq.13 for ; Tue, 25 Jun 2019 17:04:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=PV8bMP2UMNlkx7OeCEesXi507zEmJNhr1YhzchIxGFw=; b=VMcl9yOUwimfyyp1xOlNvatRARN9RV3uuCXNBJ0Ys+IAZCGEk/h7TBqIlT8JwSiqmQ LZoBbGHjFXkG+LNb1Ih0TqCHRwpMxjKZ4cewr47GfOFC3Gl+SsCzbFZh8+h8WH867g0J BZPFkux0b0V9D3mijbx0CfKGVDNQvmAwG8b7fg00D/cp8xqbe5W15shWUEY0Twnq93ZH cRVQcwWBktVo/4dvd3r5R5cTdxspTYSeC7T2RFjihsDONiOxFRVcdiweaXaIwRpWCOoe pO2e8T7disrvyeMY8OQ2Bg/LxEXaKFs81oS0wMtKgH+BWB45F/+5gU+FqDWYXgStKiC/ VSzA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: APjAAAW3a7OkZkFhmsUT45tfJwXWu9nrDzPo9tHJjBEwWc6y8DmN+HaF hMewP717WQJesXKRXl9xtG7HQZINt04QcKFiDIwfI57RikbyDV0jM7jslQB3ckLW6ymd7CGnXDT igEXy1IkX3BVaxy1GgAFYzCXf8wjdSDwJB3rio1YPbvps/Y0UQj1v9+zOj9b9U9fARA== X-Received: by 2002:a9d:6d06:: with SMTP id o6mr903790otp.225.1561507450605; Tue, 25 Jun 2019 17:04:10 -0700 (PDT) X-Google-Smtp-Source: APXvYqzR56wkTtXaqlEom5859xW2kOwvna338GM1DBXOUn9wPbz66SYChGqLPVXUqxIKmcs6dUZG X-Received: by 2002:a9d:6d06:: with SMTP id o6mr903621otp.225.1561507448071; Tue, 25 Jun 2019 17:04:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561507448; cv=none; d=google.com; s=arc-20160816; b=e87tc4+x9cU1umFmds0AkHoP0G24NEDb/JFz8KXa37A54pGmmdRC0SF8Wb1lg1ld/P ukuY9CWpc9bOlpOZVszXwFav5XjVNbP1rafCYFh+vrDfmMSyilSmAmE4tX/3S8oV0xYd FPKsk0YFI/3fCr3XdhXVRsdTijWFX1yC4xWUkxfOnIukSY1Zk/ujEjpXqieOmkrBJSS5 e7MK/fVhB3M4uJ8pcFhEfVJe9vy9TxElfEAMiu2qx61WgBAHZKUw9CMuQc5svs2BVeJo jSOv//xFs9tiboblitBfhWJNj6iM8nXq+kC06CsphFTT+y4NRj2WEg2/mXJZqRX5ljSz YKxw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=PV8bMP2UMNlkx7OeCEesXi507zEmJNhr1YhzchIxGFw=; b=IJj0ES2tLu3Febf6w1KD2lnSqsbCv/SOPgVMN/HydRNSZRGpzbXMFkuWnxa5N7Wn8T eEAPp2IcV993VhUtHbU98ZYjXJ+2phKVrjy4kstQiCSnZ5T+bq4DVtb9tlNFJWjckqNb J+RGd9WmY1nS3M84hrRmFXKUGYTE4NvyLbIB+z/fDf+mGfGIS8n4i3PNLW+xmTzqSFmP 3Nf8//tOpVZxZlnkgExyLQs0wSrMXivaHSSAK3glcyNtm1oBkpKlVy1L4BTQ1sSSeZlR LOP57GA7KgHOSF4zieWTOXvgTy0rSzsGglq2lIEaNEznlwtJ2u9Rov/cSOXyNps3U9aR V9Qw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com. [115.124.30.130]) by mx.google.com with ESMTPS id i20si8385870otr.202.2019.06.25.17.04.06 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Jun 2019 17:04:08 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.130 as permitted sender) client-ip=115.124.30.130; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01f04446;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0TVCYVJX_1561507375; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0TVCYVJX_1561507375) by smtp.aliyun-inc.com(127.0.0.1); Wed, 26 Jun 2019 08:03:03 +0800 From: Yang Shi To: kirill.shutemov@linux.intel.com, ktkhai@virtuozzo.com, hannes@cmpxchg.org, mhocko@suse.com, hughd@google.com, shakeelb@google.com, rientjes@google.com, akpm@linux-foundation.org Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 3/4] mm: shrinker: make shrinker not depend on memcg kmem Date: Wed, 26 Jun 2019 08:02:40 +0800 Message-Id: <1561507361-59349-4-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1561507361-59349-1-git-send-email-yang.shi@linux.alibaba.com> References: <1561507361-59349-1-git-send-email-yang.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently shrinker is just allocated and can work when memcg kmem is enabled. But, THP deferred split shrinker is not slab shrinker, it doesn't make too much sense to have such shrinker depend on memcg kmem. It should be able to reclaim THP even though memcg kmem is disabled. Introduce a new shrinker flag, SHRINKER_NONSLAB, for non-slab shrinker. When memcg kmem is disabled, just such shrinkers can be called in shrinking memcg slab. Cc: Kirill Tkhai Cc: Johannes Weiner Cc: Michal Hocko Cc: "Kirill A . Shutemov" Cc: Hugh Dickins Cc: Shakeel Butt Cc: David Rientjes Acked-by: Kirill A. Shutemov Signed-off-by: Yang Shi --- include/linux/shrinker.h | 3 ++- mm/vmscan.c | 36 +++++++++++++++++++----------------- 2 files changed, 21 insertions(+), 18 deletions(-) diff --git a/include/linux/shrinker.h b/include/linux/shrinker.h index 9443caf..9e112d6 100644 --- a/include/linux/shrinker.h +++ b/include/linux/shrinker.h @@ -69,7 +69,7 @@ struct shrinker { /* These are for internal use */ struct list_head list; -#ifdef CONFIG_MEMCG_KMEM +#ifdef CONFIG_MEMCG /* ID in shrinker_idr */ int id; #endif @@ -81,6 +81,7 @@ struct shrinker { /* Flags */ #define SHRINKER_NUMA_AWARE (1 << 0) #define SHRINKER_MEMCG_AWARE (1 << 1) +#define SHRINKER_NONSLAB (1 << 2) extern int prealloc_shrinker(struct shrinker *shrinker); extern void register_shrinker_prepared(struct shrinker *shrinker); diff --git a/mm/vmscan.c b/mm/vmscan.c index 7889f58..187cacb 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -174,8 +174,7 @@ struct scan_control { static LIST_HEAD(shrinker_list); static DECLARE_RWSEM(shrinker_rwsem); -#ifdef CONFIG_MEMCG_KMEM - +#ifdef CONFIG_MEMCG /* * We allow subsystems to populate their shrinker-related * LRU lists before register_shrinker_prepared() is called @@ -227,18 +226,7 @@ static void unregister_memcg_shrinker(struct shrinker *shrinker) idr_remove(&shrinker_idr, id); up_write(&shrinker_rwsem); } -#else /* CONFIG_MEMCG_KMEM */ -static int prealloc_memcg_shrinker(struct shrinker *shrinker) -{ - return 0; -} -static void unregister_memcg_shrinker(struct shrinker *shrinker) -{ -} -#endif /* CONFIG_MEMCG_KMEM */ - -#ifdef CONFIG_MEMCG static bool global_reclaim(struct scan_control *sc) { return !sc->target_mem_cgroup; @@ -293,6 +281,15 @@ static bool memcg_congested(pg_data_t *pgdat, } #else +static int prealloc_memcg_shrinker(struct shrinker *shrinker) +{ + return 0; +} + +static void unregister_memcg_shrinker(struct shrinker *shrinker) +{ +} + static bool global_reclaim(struct scan_control *sc) { return true; @@ -579,7 +576,7 @@ static unsigned long do_shrink_slab(struct shrink_control *shrinkctl, return freed; } -#ifdef CONFIG_MEMCG_KMEM +#ifdef CONFIG_MEMCG static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, int priority) { @@ -587,7 +584,7 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, unsigned long ret, freed = 0; int i; - if (!memcg_kmem_enabled() || !mem_cgroup_online(memcg)) + if (!mem_cgroup_online(memcg)) return 0; if (!down_read_trylock(&shrinker_rwsem)) @@ -613,6 +610,11 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, continue; } + /* Call non-slab shrinkers even though kmem is disabled */ + if (!memcg_kmem_enabled() && + !(shrinker->flags & SHRINKER_NONSLAB)) + continue; + ret = do_shrink_slab(&sc, shrinker, priority); if (ret == SHRINK_EMPTY) { clear_bit(i, map->map); @@ -649,13 +651,13 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, up_read(&shrinker_rwsem); return freed; } -#else /* CONFIG_MEMCG_KMEM */ +#else /* CONFIG_MEMCG */ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, int priority) { return 0; } -#endif /* CONFIG_MEMCG_KMEM */ +#endif /* CONFIG_MEMCG */ /** * shrink_slab - shrink slab caches From patchwork Wed Jun 26 00:02:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Shi X-Patchwork-Id: 11016645 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5D49A14BB for ; Wed, 26 Jun 2019 00:04:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4D3F1285D2 for ; Wed, 26 Jun 2019 00:04:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 41603285DD; Wed, 26 Jun 2019 00:04:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 601DC285D2 for ; Wed, 26 Jun 2019 00:04:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D1056B0006; Tue, 25 Jun 2019 20:04:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6A6088E0003; Tue, 25 Jun 2019 20:04:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 570788E0002; Tue, 25 Jun 2019 20:04:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by kanga.kvack.org (Postfix) with ESMTP id 207FC6B0006 for ; Tue, 25 Jun 2019 20:04:15 -0400 (EDT) Received: by mail-pl1-f198.google.com with SMTP id p14so228858plq.1 for ; Tue, 25 Jun 2019 17:04:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=S797eYnzFdTMMfjZJVlwmlDvblrr20h1Xd/x4lcf5OU=; b=riR+0wkK7nhWCJIwRFfK/0fWlly4F5MJdSLeGNqJ3jywC9f6hFsnxg20sIEr54PSsg nfGC5v2QasUPl5AxsAqXHPgnzOYok1cet7txSBEHqtH5sHoJHpfXv6yFFRJPzTAHe940 GLb3Z3z5VoCxMQ0vN+A9qh2IBHskZpEI7/xUoReycGPkRgX1aQyMxEpVhxXMVDbQpkQw tY4ue4zRlusz/SkQH5FmmNXLspRtyhhEDyYRXSjT/rpXOw5Ak6V6iZctPqX5Zgp5rnJM 5v6BaaJntLV1o0qN7+uG+RzpjJqJB/cccJzvYyJQkQRkzgQH+JLaXw8TDnSb6O+ncISs oTkg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.37 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Gm-Message-State: APjAAAXKZIPMoAl4G7mTlHfxQIDdjj/T04Bl0B+qfAKXZl2qZ0GBNDOw rObEXSw6QL6aYCx5pNkW5Hjx5zLqinI2YfJuSpLu2SHyxYB7ZFyavMbYVOeyFWEIggan5lJe7ew Qpp/CgZukjRuv3PuDfRD47r5sK1CTqbE5uuWb70i8Nrpln4cO1pSsr6OxxE8sOq0tag== X-Received: by 2002:a17:90a:360c:: with SMTP id s12mr704058pjb.30.1561507454736; Tue, 25 Jun 2019 17:04:14 -0700 (PDT) X-Google-Smtp-Source: APXvYqwm/blPXQZO+JfBS61gPdg8MmNlVLFvlwYjJUYSXc8HOQFZrIaV7aof5RcY8Q0WQRg/UfOJ X-Received: by 2002:a17:90a:360c:: with SMTP id s12mr703929pjb.30.1561507453259; Tue, 25 Jun 2019 17:04:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1561507453; cv=none; d=google.com; s=arc-20160816; b=oVYUEYqxwLnwlwHIfd9oVxn4i9i1F8KHjq372TO9fnUxbNgh68py/IT/dBH1eZmWjT GFi6o+SDagKzuxqslDagyPoHb3BCo0oVgKEuM94dHsov538jpOSyvl9QSDurdJ4ICv3N JizR4Q2E/PDouyWeWnD5Q/1LL7LzlHacMACoysmpS/8EQPqpykZc/NV4k0vBylXxOd1J X4Yl0n/wB5X8aMQGfOp9GWNYf5beRnadjdShUSxUjZDdOSXGgWnXqhK7WHmuMIsiOlkj OSJnPVThSDqT/p53LNBb3wulBPNIHLwbJqH+OVxHvYRW5k72zxbHkr42lnRG3rW5s7wq Lu5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from; bh=S797eYnzFdTMMfjZJVlwmlDvblrr20h1Xd/x4lcf5OU=; b=xTTVC7PD+tA4goR5Z5of/ATRnUUsRCh0MaMTCe1HThsM4ctK8A402UCGa0bN0wX0vj 6WpbT7tlQY/vg6+NJTk3yb15A1pMZBrxP39apmpLVa4SXFP4MphdifaV4XdHp47cljKt XipR5CcdV5SjZ4kmDI6hL94xRCr9qMQE74ozgg+fp/28NjejL4CQf20tiOqVFKThkp3i idjNU1EA7hEquxNWZDZ1jAbqCvPQ7ZKGE0aZQJCwFGVYTWU4jAFY04B1FU4BdHbCnq1U q0WATpnWz/TIzcqBtPv5JaRHSI/IMhysYmZzJcaIDx0VUi/B6mRyZw2UNbGmCPNK+6xj 6Ttw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.37 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com Received: from out4437.biz.mail.alibaba.com (out4437.biz.mail.alibaba.com. [47.88.44.37]) by mx.google.com with ESMTPS id y135si10968724pfg.200.2019.06.25.17.04.11 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 25 Jun 2019 17:04:13 -0700 (PDT) Received-SPF: pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.37 as permitted sender) client-ip=47.88.44.37; Authentication-Results: mx.google.com; spf=pass (google.com: domain of yang.shi@linux.alibaba.com designates 47.88.44.37 as permitted sender) smtp.mailfrom=yang.shi@linux.alibaba.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=yang.shi@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0TVCYVJX_1561507375; Received: from e19h19392.et15sqa.tbsite.net(mailfrom:yang.shi@linux.alibaba.com fp:SMTPD_---0TVCYVJX_1561507375) by smtp.aliyun-inc.com(127.0.0.1); Wed, 26 Jun 2019 08:03:03 +0800 From: Yang Shi To: kirill.shutemov@linux.intel.com, ktkhai@virtuozzo.com, hannes@cmpxchg.org, mhocko@suse.com, hughd@google.com, shakeelb@google.com, rientjes@google.com, akpm@linux-foundation.org Cc: yang.shi@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [v4 PATCH 4/4] mm: thp: make deferred split shrinker memcg aware Date: Wed, 26 Jun 2019 08:02:41 +0800 Message-Id: <1561507361-59349-5-git-send-email-yang.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1561507361-59349-1-git-send-email-yang.shi@linux.alibaba.com> References: <1561507361-59349-1-git-send-email-yang.shi@linux.alibaba.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Currently THP deferred split shrinker is not memcg aware, this may cause premature OOM with some configuration. For example the below test would run into premature OOM easily: $ cgcreate -g memory:thp $ echo 4G > /sys/fs/cgroup/memory/thp/memory/limit_in_bytes $ cgexec -g memory:thp transhuge-stress 4000 transhuge-stress comes from kernel selftest. It is easy to hit OOM, but there are still a lot THP on the deferred split queue, memcg direct reclaim can't touch them since the deferred split shrinker is not memcg aware. Convert deferred split shrinker memcg aware by introducing per memcg deferred split queue. The THP should be on either per node or per memcg deferred split queue if it belongs to a memcg. When the page is immigrated to the other memcg, it will be immigrated to the target memcg's deferred split queue too. Reuse the second tail page's deferred_list for per memcg list since the same THP can't be on multiple deferred split queues. Cc: Kirill Tkhai Cc: Johannes Weiner Cc: Michal Hocko Cc: "Kirill A . Shutemov" Cc: Hugh Dickins Cc: Shakeel Butt Cc: David Rientjes Acked-by: Kirill A. Shutemov Signed-off-by: Yang Shi --- include/linux/huge_mm.h | 9 +++++++ include/linux/memcontrol.h | 4 +++ include/linux/mm_types.h | 1 + mm/huge_memory.c | 63 ++++++++++++++++++++++++++++++++++++++-------- mm/memcontrol.c | 24 ++++++++++++++++++ 5 files changed, 90 insertions(+), 11 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 7cd5c15..7738509 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -250,6 +250,15 @@ static inline bool thp_migration_supported(void) return IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION); } +static inline struct list_head *page_deferred_list(struct page *page) +{ + /* + * Global or memcg deferred list in the second tail pages is + * occupied by compound_head. + */ + return &page[2].deferred_list; +} + #else /* CONFIG_TRANSPARENT_HUGEPAGE */ #define HPAGE_PMD_SHIFT ({ BUILD_BUG(); 0; }) #define HPAGE_PMD_MASK ({ BUILD_BUG(); 0; }) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 1dcb763..77a313c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -311,6 +311,10 @@ struct mem_cgroup { struct list_head event_list; spinlock_t event_list_lock; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + struct deferred_split deferred_split_queue; +#endif + struct mem_cgroup_per_node *nodeinfo[0]; /* WARNING: nodeinfo must be the last member here */ }; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8ec38b1..4eabf80 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -139,6 +139,7 @@ struct page { struct { /* Second tail page of compound page */ unsigned long _compound_pad_1; /* compound_head */ unsigned long _compound_pad_2; + /* For both global and memcg */ struct list_head deferred_list; }; struct { /* Page table pages */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 81cf759..b00570d 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -492,11 +492,25 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) return pmd; } -static inline struct list_head *page_deferred_list(struct page *page) +#ifdef CONFIG_MEMCG +static inline struct deferred_split *get_deferred_split_queue(struct page *page) { - /* ->lru in the tail pages is occupied by compound_head. */ - return &page[2].deferred_list; + struct mem_cgroup *memcg = compound_head(page)->mem_cgroup; + struct pglist_data *pgdat = NODE_DATA(page_to_nid(page)); + + if (memcg) + return &memcg->deferred_split_queue; + else + return &pgdat->deferred_split_queue; } +#else +static inline struct deferred_split *get_deferred_split_queue(struct page *page) +{ + struct pglist_data *pgdat = NODE_DATA(page_to_nid(page)); + + return &pgdat->deferred_split_queue; +} +#endif void prep_transhuge_page(struct page *page) { @@ -2658,7 +2672,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) { struct page *head = compound_head(page); struct pglist_data *pgdata = NODE_DATA(page_to_nid(head)); - struct deferred_split *ds_queue = &pgdata->deferred_split_queue; + struct deferred_split *ds_queue = get_deferred_split_queue(page); struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; int count, mapcount, extra_pins, ret; @@ -2794,8 +2808,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) void free_transhuge_page(struct page *page) { - struct pglist_data *pgdata = NODE_DATA(page_to_nid(page)); - struct deferred_split *ds_queue = &pgdata->deferred_split_queue; + struct deferred_split *ds_queue = get_deferred_split_queue(page); unsigned long flags; spin_lock_irqsave(&ds_queue->split_queue_lock, flags); @@ -2809,8 +2822,10 @@ void free_transhuge_page(struct page *page) void deferred_split_huge_page(struct page *page) { - struct pglist_data *pgdata = NODE_DATA(page_to_nid(page)); - struct deferred_split *ds_queue = &pgdata->deferred_split_queue; + struct deferred_split *ds_queue = get_deferred_split_queue(page); +#ifdef CONFIG_MEMCG + struct mem_cgroup *memcg = compound_head(page)->mem_cgroup; +#endif unsigned long flags; VM_BUG_ON_PAGE(!PageTransHuge(page), page); @@ -2820,6 +2835,11 @@ void deferred_split_huge_page(struct page *page) count_vm_event(THP_DEFERRED_SPLIT_PAGE); list_add_tail(page_deferred_list(page), &ds_queue->split_queue); ds_queue->split_queue_len++; +#ifdef CONFIG_MEMCG + if (memcg) + memcg_set_shrinker_bit(memcg, page_to_nid(page), + deferred_split_shrinker.id); +#endif } spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } @@ -2827,8 +2847,19 @@ void deferred_split_huge_page(struct page *page) static unsigned long deferred_split_count(struct shrinker *shrink, struct shrink_control *sc) { + struct deferred_split *ds_queue; struct pglist_data *pgdata = NODE_DATA(sc->nid); - struct deferred_split *ds_queue = &pgdata->deferred_split_queue; + +#ifdef CONFIG_MEMCG + if (!sc->memcg) { + ds_queue = &pgdata->deferred_split_queue; + return READ_ONCE(ds_queue->split_queue_len); + } + + ds_queue = &sc->memcg->deferred_split_queue; +#else + ds_queue = &pgdata->deferred_split_queue; +#endif return READ_ONCE(ds_queue->split_queue_len); } @@ -2836,12 +2867,21 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc) { struct pglist_data *pgdata = NODE_DATA(sc->nid); - struct deferred_split *ds_queue = &pgdata->deferred_split_queue; + struct deferred_split *ds_queue; unsigned long flags; LIST_HEAD(list), *pos, *next; struct page *page; int split = 0; +#ifdef CONFIG_MEMCG + if (sc->memcg) + ds_queue = &sc->memcg->deferred_split_queue; + else + ds_queue = &pgdata->deferred_split_queue; +#else + ds_queue = &pgdata->deferred_split_queue; +#endif + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); /* Take pin on all head pages to avoid freeing them under us */ list_for_each_safe(pos, next, &ds_queue->split_queue) { @@ -2888,7 +2928,8 @@ static unsigned long deferred_split_scan(struct shrinker *shrink, .count_objects = deferred_split_count, .scan_objects = deferred_split_scan, .seeks = DEFAULT_SEEKS, - .flags = SHRINKER_NUMA_AWARE, + .flags = SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE | + SHRINKER_NONSLAB, }; #ifdef CONFIG_DEBUG_FS diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ba9138a..5730608 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4585,6 +4585,11 @@ static struct mem_cgroup *mem_cgroup_alloc(void) #ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD(&memcg->cgwb_list); #endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + spin_lock_init(&memcg->deferred_split_queue.split_queue_lock); + INIT_LIST_HEAD(&memcg->deferred_split_queue.split_queue); + memcg->deferred_split_queue.split_queue_len = 0; +#endif idr_replace(&mem_cgroup_idr, memcg, memcg->id.id); return memcg; fail: @@ -4955,6 +4960,14 @@ static int mem_cgroup_move_account(struct page *page, __mod_memcg_state(to, NR_WRITEBACK, nr_pages); } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (compound && !list_empty(page_deferred_list(page))) { + spin_lock(&from->deferred_split_queue.split_queue_lock); + list_del_init(page_deferred_list(page)); + from->deferred_split_queue.split_queue_len--; + spin_unlock(&from->deferred_split_queue.split_queue_lock); + } +#endif /* * It is safe to change page->mem_cgroup here because the page * is referenced, charged, and isolated - we can't race with @@ -4963,6 +4976,17 @@ static int mem_cgroup_move_account(struct page *page, /* caller should have done css_get */ page->mem_cgroup = to; + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (compound && list_empty(page_deferred_list(page))) { + spin_lock(&to->deferred_split_queue.split_queue_lock); + list_add_tail(page_deferred_list(page), + &to->deferred_split_queue.split_queue); + to->deferred_split_queue.split_queue_len++; + spin_unlock(&to->deferred_split_queue.split_queue_lock); + } +#endif + spin_unlock_irqrestore(&from->move_lock, flags); ret = 0;