From patchwork Fri May 18 04:29:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: TSUKADA Koutaro X-Patchwork-Id: 10408169 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8E3E5601F9 for ; Fri, 18 May 2018 04:29:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E183284FF for ; Fri, 18 May 2018 04:29:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 71408285D9; Fri, 18 May 2018 04:29:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 90346284FF for ; Fri, 18 May 2018 04:29:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FA4D6B056A; Fri, 18 May 2018 00:29:54 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9A98F6B056B; Fri, 18 May 2018 00:29:54 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C26E6B056C; Fri, 18 May 2018 00:29:54 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f69.google.com (mail-pl0-f69.google.com [209.85.160.69]) by kanga.kvack.org (Postfix) with ESMTP id 480C26B056A for ; Fri, 18 May 2018 00:29:54 -0400 (EDT) Received: by mail-pl0-f69.google.com with SMTP id a14-v6so4300883plt.7 for ; Thu, 17 May 2018 21:29:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:subject:to:cc :references:from:message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=taNWzUa6U4Gh2hjgPaDlLfOeIfjDDidZbAGEPoUw9y8=; b=U974jelJeb7q5E7UppUARn9ylpkcGcM6MReELzp0+VjHmMA4KfYJjo48OSPHjffWDy uLo8i8KgEXkIF5P1fNNrC4h9uk/UMjpqqBfB8GceiAK54K9t0bkrePKfeHwxtksVnCIT hTCRbDHUdeyiu3q/aTTU6b1Bl5qfWl7d9RCsNO+A+2MOKZoynsFZfgLvRWJSlguF5Axk XhZgJNmg+ViqYW68WmS11hVjTjSIHzvmGEQRqmeb6lPdZipOo5qoWPqHMX43AZwD2r8c YOsM5LqaNjAFP4/GJKLPf7ZzGt1focadtmjzq+2UJy0XOdRotYWti/6UDcoZc6AnvpDB Yiyg== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of tsukada@ascade.co.jp designates 218.224.228.194 as permitted sender) smtp.mailfrom=tsukada@ascade.co.jp X-Gm-Message-State: ALKqPwf6wIIpS92o7YKxtYCiBVYsqy9Ut5MfTbbeW8bMjrZz7ngEiS0H fFPnurgA6yQbIc0YbnZi2ttmufXQtqRyb1bEF0mwr/OVCWz8RDh0RKsC+en6XtTbPlK0yrJWQYG +arWEHxzvn4KAgHV3yUEbcq8Wuz+culQ6L6P77lZGt+O4XQTE9zLuhCWseo9NBLa6oQ== X-Received: by 2002:a17:902:8f84:: with SMTP id z4-v6mr8039237plo.194.1526617793981; Thu, 17 May 2018 21:29:53 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpOZY0yhdX7BiP6leWGm0blG3LmJg9/yxR4cBvTwN3XulMLER2nXLDsiofhBv8oNl9QteD9 X-Received: by 2002:a17:902:8f84:: with SMTP id z4-v6mr8039210plo.194.1526617793275; Thu, 17 May 2018 21:29:53 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1526617793; cv=none; d=google.com; s=arc-20160816; b=kLhh57+NOm1gI7eL8B2MGAQhokxeIB3JjEE8CJKH3Z/ePHQLoatTvDo/bFbxzhynox DKo1IDC9Tyux88pXeArQe0KRBMd6DnSsspj8Kc0OpmGrUyYQgellai0HSiL7IuVsAw1w La5BH6SBiHyaFnNfQSOOf4xjyZocVQcyV5jb4nNG5ex4cmWa9HzBCnSJBihWLw8oqTYk mai7BaOVAuB1zvj8rJYmaIliTzEfzhuEd7tSrlqCXBggL4VgvGmVIkil1gN7MujRFpb+ hVzAYT4aCvU7zS4Fj9lpVhFCWB3Jib10zvdwsB3NravRaJKEV4afTvO4HS/tyMzmga7M Q6oA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:content-language:in-reply-to:mime-version :user-agent:date:message-id:from:references:cc:to:subject :arc-authentication-results; bh=taNWzUa6U4Gh2hjgPaDlLfOeIfjDDidZbAGEPoUw9y8=; b=bhbjpGnI7JnCiWKArYHhcxX1wX7WD63NoTwJ+nCWnq7uj4XXZVGC34rD3NEOkdrCEY SrkfUaH8xLHy2k5H60RQhMniyY7VGnAq/kPK11wpypS39kZfYmmOO3Jn+c84i5zSb/cN fPNpBiNDD8Jah8airSgTUwXUUUBp+Q49kGWe6ZR6OjgvCrZd+eyEWO3M2b6I6p3Is8oo zElkjelWMllj1/btffPNmtW0Ke7bSzr9FH94GrfKfZ3NAQOX8kT0eWQg+PvTWwFhgN4i qlhBs2bV+otPcz/rBzEuKNyEfGcPLvqATewPnDWCs1rIFZ3KBrHShTCRYF7ClMo58zYi cKsQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of tsukada@ascade.co.jp designates 218.224.228.194 as permitted sender) smtp.mailfrom=tsukada@ascade.co.jp Received: from ns.ascade.co.jp (ext-host0001.ascade.co.jp. [218.224.228.194]) by mx.google.com with ESMTP id e92-v6si6440102pld.601.2018.05.17.21.29.52 for ; Thu, 17 May 2018 21:29:53 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of tsukada@ascade.co.jp designates 218.224.228.194 as permitted sender) client-ip=218.224.228.194; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of tsukada@ascade.co.jp designates 218.224.228.194 as permitted sender) smtp.mailfrom=tsukada@ascade.co.jp Received: from server0001.ascade.co.jp (server0001.ascade.co.jp [10.1.1.63]) by ns.ascade.co.jp (Postfix) with ESMTP id 062DC993CF; Fri, 18 May 2018 13:29:52 +0900 (JST) Received: from [IPv6:::1] (server0001.ascade.co.jp [10.1.1.63]) by server0001.ascade.co.jp (Postfix) with ESMTP id 9BBA91004E1; Fri, 18 May 2018 13:29:50 +0900 (JST) Subject: [PATCH v2 1/7] hugetlb: introduce charge_surplus_huge_pages to struct hstate To: Johannes Weiner , Michal Hocko , Vladimir Davydov , Jonathan Corbet , "Luis R. Rodriguez" , Kees Cook Cc: Andrew Morton , Roman Gushchin , David Rientjes , Mike Kravetz , "Aneesh Kumar K.V" , Naoya Horiguchi , Anshuman Khandual , Marc-Andre Lureau , Punit Agrawal , Dan Williams , Vlastimil Babka , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, tsukada@ascade.co.jp References: From: TSUKADA Koutaro Message-ID: Date: Fri, 18 May 2018 13:29:43 +0900 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The charge_surplus_huge_pages indicates to charge surplus huge pages obteined from the normal page pool to memory cgroup. The default value is false. This patch implements the core part of charging surplus hugepages. Use the private and mem_cgroup member of the second entry of compound hugepage for surplus hugepage charging. Mark when surplus hugepage is obtained from normal pool, and charge to memory cgroup at alloc_huge_page. Once the mapping of the page is decided, commit the charge. surplus hugepages will uncharge or cancel at free_huge_page. Signed-off-by: TSUKADA Koutaro --- include/linux/hugetlb.h | 2 mm/hugetlb.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 102 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 36fa6a2..33fe5be 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -158,6 +158,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot); bool is_hugetlb_entry_migration(pte_t pte); +bool PageSurplusCharge(struct page *page); #else /* !CONFIG_HUGETLB_PAGE */ @@ -338,6 +339,7 @@ struct hstate { unsigned int nr_huge_pages_node[MAX_NUMNODES]; unsigned int free_huge_pages_node[MAX_NUMNODES]; unsigned int surplus_huge_pages_node[MAX_NUMNODES]; + bool charge_surplus_huge_pages; /* default to off */ #ifdef CONFIG_CGROUP_HUGETLB /* cgroup control files */ struct cftype cgroup_files[5]; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 2186791..679c151f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -36,6 +36,7 @@ #include #include #include +#include #include "internal.h" int hugetlb_max_hstate __read_mostly; @@ -1236,6 +1237,90 @@ static inline void ClearPageHugeTemporary(struct page *page) page[2].mapping = NULL; } +#define HUGETLB_SURPLUS_CHARGE 1UL + +bool PageSurplusCharge(struct page *page) +{ + if (!PageHuge(page)) + return false; + return page[1].private == HUGETLB_SURPLUS_CHARGE; +} + +static inline void SetPageSurplusCharge(struct page *page) +{ + page[1].private = HUGETLB_SURPLUS_CHARGE; +} + +static inline void ClearPageSurplusCharge(struct page *page) +{ + page[1].private = 0; +} + +static inline void +set_surplus_hugepage_memcg(struct page *page, struct mem_cgroup *memcg) +{ + page[1].mem_cgroup = memcg; +} + +static inline struct mem_cgroup *get_surplus_hugepage_memcg(struct page *page) +{ + return page[1].mem_cgroup; +} + +static void surplus_hugepage_set_charge(struct hstate *h, struct page *page) +{ + if (likely(!h->charge_surplus_huge_pages)) + return; + if (unlikely(!page)) + return; + SetPageSurplusCharge(page); +} + +static int surplus_hugepage_try_charge(struct page *page, struct mm_struct *mm) +{ + struct mem_cgroup *memcg; + + if (likely(!PageSurplusCharge(page))) + return 0; + + if (mem_cgroup_try_charge(page, mm, GFP_KERNEL, &memcg, true)) { + /* mem_cgroup oom invoked */ + ClearPageSurplusCharge(page); + return -ENOMEM; + } + set_surplus_hugepage_memcg(page, memcg); + + return 0; +} + +static void surplus_hugepage_commit_charge(struct page *page) +{ + struct mem_cgroup *memcg; + + if (likely(!PageSurplusCharge(page))) + return; + + memcg = get_surplus_hugepage_memcg(page); + mem_cgroup_commit_charge(page, memcg, false, true); + set_surplus_hugepage_memcg(page, NULL); +} + +static void surplus_hugepage_finalize_charge(struct page *page) +{ + struct mem_cgroup *memcg; + + if (likely(!PageSurplusCharge(page))) + return; + + memcg = get_surplus_hugepage_memcg(page); + if (memcg) + mem_cgroup_cancel_charge(page, memcg, true); + else + mem_cgroup_uncharge(page); + set_surplus_hugepage_memcg(page, NULL); + ClearPageSurplusCharge(page); +} + void free_huge_page(struct page *page) { /* @@ -1248,6 +1333,8 @@ void free_huge_page(struct page *page) (struct hugepage_subpool *)page_private(page); bool restore_reserve; + surplus_hugepage_finalize_charge(page); + set_page_private(page, 0); page->mapping = NULL; VM_BUG_ON_PAGE(page_count(page), page); @@ -1583,6 +1670,8 @@ static struct page *alloc_surplus_huge_page(struct hstate *h, gfp_t gfp_mask, out_unlock: spin_unlock(&hugetlb_lock); + surplus_hugepage_set_charge(h, page); + return page; } @@ -2062,6 +2151,11 @@ struct page *alloc_huge_page(struct vm_area_struct *vma, hugetlb_cgroup_commit_charge(idx, pages_per_huge_page(h), h_cg, page); spin_unlock(&hugetlb_lock); + if (unlikely(surplus_hugepage_try_charge(page, vma->vm_mm))) { + put_page(page); + return ERR_PTR(-ENOMEM); + } + set_page_private(page, (unsigned long)spool); map_commit = vma_commit_reservation(h, vma, addr); @@ -3610,6 +3704,7 @@ static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma, make_huge_pte(vma, new_page, 1)); page_remove_rmap(old_page, true); hugepage_add_new_anon_rmap(new_page, vma, address); + surplus_hugepage_commit_charge(new_page); /* Make the old page be freed below */ new_page = old_page; } @@ -3667,6 +3762,9 @@ int huge_add_to_page_cache(struct page *page, struct address_space *mapping, if (err) return err; + + surplus_hugepage_commit_charge(page); + ClearPagePrivate(page); spin_lock(&inode->i_lock); @@ -3809,6 +3907,7 @@ static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma, if (anon_rmap) { ClearPagePrivate(page); hugepage_add_new_anon_rmap(page, vma, address); + surplus_hugepage_commit_charge(page); } else page_dup_rmap(page, true); new_pte = make_huge_pte(vma, page, ((vma->vm_flags & VM_WRITE) @@ -4108,6 +4207,7 @@ int hugetlb_mcopy_atomic_pte(struct mm_struct *dst_mm, } else { ClearPagePrivate(page); hugepage_add_new_anon_rmap(page, dst_vma, dst_addr); + surplus_hugepage_commit_charge(page); } _dst_pte = make_huge_pte(dst_vma, page, dst_vma->vm_flags & VM_WRITE);