From patchwork Wed Oct 30 01:36:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mina Almasry X-Patchwork-Id: 11218917 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35F841599 for ; Wed, 30 Oct 2019 01:37:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E6C5821835 for ; Wed, 30 Oct 2019 01:37:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="IoXoUJup" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E6C5821835 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 535C66B0008; Tue, 29 Oct 2019 21:37:15 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4E8F66B000A; Tue, 29 Oct 2019 21:37:15 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 389D36B000C; Tue, 29 Oct 2019 21:37:15 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0044.hostedemail.com [216.40.44.44]) by kanga.kvack.org (Postfix) with ESMTP id 05C4F6B0008 for ; Tue, 29 Oct 2019 21:37:14 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with SMTP id 959104DDA for ; Wed, 30 Oct 2019 01:37:14 +0000 (UTC) X-FDA: 76098737988.01.glove01_3f2fdafdf1030 X-Spam-Summary: 2,0,0,cb94d5ca75dcdc32,d41d8cd98f00b204,3sem4xqskcpiufgumlsgchuaiiafy.wigfchor-ggepuwe.ila@flex--almasrymina.bounces.google.com,:mike.kravetz@oracle.com:shuah@kernel.org:almasrymina@google.com:linux-kernel@vger.kernel.org::linux-kselftest@vger.kernel.org:cgroups@vger.kernel.org:aneesh.kumar@linux.vnet.ibm.com:hdanton@sina.com,RULES_HIT:2:41:69:152:355:379:541:800:960:966:973:988:989:1260:1277:1313:1314:1345:1359:1431:1437:1516:1518:1535:1593:1594:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2559:2562:2693:2890:2902:3138:3139:3140:3141:3142:3152:3865:3866:3867:3868:3870:3871:3872:3874:4042:4049:4120:4321:4385:4605:5007:6119:6261:6653:7875:7903:9036:9108:9592:9969:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12555:12683:12895:12986:13161:13229:14394:14659:21080:21444:21450:21451:21627:21789:21939:30001:30012:30054:30070,0,RBL:209.85.161.74:@flex--almasrymina.bounces.google.com:.lbl8.mailshell.net-62.18.175.100 66.100.201.100,CacheIP :none,Ba X-HE-Tag: glove01_3f2fdafdf1030 X-Filterd-Recvd-Size: 9199 Received: from mail-yw1-f74.google.com (mail-yw1-f74.google.com [209.85.161.74]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Wed, 30 Oct 2019 01:37:14 +0000 (UTC) Received: by mail-yw1-f74.google.com with SMTP id r64so533102ywb.3 for ; Tue, 29 Oct 2019 18:37:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qPaHEbPrYv6JUTplbHxnlfnp4zAORaH8vrWi/UicSm4=; b=IoXoUJuphbNgV9+X/Lt6knalFErc6s3NPqkOlQ1S9+wbHCVkgdrrVTFFV8e96qVxr/ 0WxOP+lLX3dm04M/Rm4QLxaw7PPDMOb4YA8sQ4FqlOteYVc4hLtgPZ8beJub3SoPjyJr AAWsxEFMqvfgjw7+6q9NT6dyDAVWFGcoldP5pC6gP6HhqcJRehTktAk+nJJSaGPbX3GK hLIFa8uM4Gei6GarKIYl6WV2534SqhVj2JpP9eQZRGef5nz6hXPnqrtlA59DHOmXOfQd VmNuRLkgOzTGa91wU/Dkt/tAuEfv4kgUBdrP141WQYWSiuXeHYv58sO8Llt+5ADfAKsF LPHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qPaHEbPrYv6JUTplbHxnlfnp4zAORaH8vrWi/UicSm4=; b=LSFZ54Ss7zNbx2qcg9xZ+QpmK3w/ufep5eu/0/8YNZtlKfOOYf/TnwfYvB4YFhMU8h xpyoNaxW7r54bjKLqOIrHcVmMKRWoH5Ujs9/Xm/jZO2u8vYHEt4CV8ODPOwfV9M3pb/O WvSwSs4SVp295j/7T3Z4mDg79oK1CC4z/ShB+lAFKA6xL7MIoX107Kq2e9tW5SX7yddF s006ehZq0w36JS2SWwY4+IF9/3tgifvI18ldpsJyMPbxoP1oSGN9rAywCzqnyail1t02 Boh+vfZLzcOB1LHjHYiq8bc8+MwpfmIVZWPcff3FlFKQa3L80/uw3cOR/b20pVb1kzA4 e12g== X-Gm-Message-State: APjAAAXDR3kUcK1BxH90BHQQ1fTA6z+IXt2BK3oy9vCQCBzosCtg2fmG qLAJLObeiCpM6q9peBaYMmObvolwwg6oLoM/JA== X-Google-Smtp-Source: APXvYqwq6yygYVt3tY6tYtpBGDJ/fDbgkOJS1xGErEBdHsCzNqN+627UxywOTbJIshRr0GqN+Ey54sxWbBYsAt3Bog== X-Received: by 2002:a81:25d7:: with SMTP id l206mr19187868ywl.36.1572399433419; Tue, 29 Oct 2019 18:37:13 -0700 (PDT) Date: Tue, 29 Oct 2019 18:36:56 -0700 In-Reply-To: <20191030013701.39647-1-almasrymina@google.com> Message-Id: <20191030013701.39647-4-almasrymina@google.com> Mime-Version: 1.0 References: <20191030013701.39647-1-almasrymina@google.com> X-Mailer: git-send-email 2.24.0.rc1.363.gb1bccd3e3d-goog Subject: [PATCH v8 4/9] hugetlb_cgroup: add reservation accounting for private mappings From: Mina Almasry To: mike.kravetz@oracle.com Cc: shuah@kernel.org, almasrymina@google.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, cgroups@vger.kernel.org, aneesh.kumar@linux.vnet.ibm.com, Hillf Danton X-Bogosity: Ham, tests=bogofilter, spamicity=0.000009, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Normally the pointer to the cgroup to uncharge hangs off the struct page, and gets queried when it's time to free the page. With hugetlb_cgroup reservations, this is not possible. Because it's possible for a page to be reserved by one task and actually faulted in by another task. The best place to put the hugetlb_cgroup pointer to uncharge for reservations is in the resv_map. But, because the resv_map has different semantics for private and shared mappings, the code patch to charge/uncharge shared and private mappings is different. This patch implements charging and uncharging for private mappings. For private mappings, the counter to uncharge is in resv_map->reservation_counter. On initializing the resv_map this is set to NULL. On reservation of a region in private mapping, the tasks hugetlb_cgroup is charged and the hugetlb_cgroup is placed is resv_map->reservation_counter. On hugetlb_vm_op_close, we uncharge resv_map->reservation_counter. Signed-off-by: Mina Almasry Acked-by: Hillf Danton --- include/linux/hugetlb.h | 8 +++++++ include/linux/hugetlb_cgroup.h | 11 +++++++++ mm/hugetlb.c | 44 +++++++++++++++++++++++++++++++++- mm/hugetlb_cgroup.c | 12 ---------- 4 files changed, 62 insertions(+), 13 deletions(-) -- 2.24.0.rc1.363.gb1bccd3e3d-goog diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 9c49a0ba894d3..36dcda7be4b0e 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -46,6 +46,14 @@ struct resv_map { long adds_in_progress; struct list_head region_cache; long region_cache_count; +#ifdef CONFIG_CGROUP_HUGETLB + /* + * On private mappings, the counter to uncharge reservations is stored + * here. If these fields are 0, then the mapping is shared. + */ + struct page_counter *reservation_counter; + unsigned long pages_per_hpage; +#endif }; extern struct resv_map *resv_map_alloc(void); void resv_map_release(struct kref *ref); diff --git a/include/linux/hugetlb_cgroup.h b/include/linux/hugetlb_cgroup.h index 1bb58a63af586..f6e3d74a02536 100644 --- a/include/linux/hugetlb_cgroup.h +++ b/include/linux/hugetlb_cgroup.h @@ -25,6 +25,17 @@ struct hugetlb_cgroup; #define HUGETLB_CGROUP_MIN_ORDER 3 #ifdef CONFIG_CGROUP_HUGETLB +struct hugetlb_cgroup { + struct cgroup_subsys_state css; + /* + * the counter to account for hugepages from hugetlb. + */ + struct page_counter hugepage[HUGE_MAX_HSTATE]; + /* + * the counter to account for hugepage reservations from hugetlb. + */ + struct page_counter reserved_hugepage[HUGE_MAX_HSTATE]; +}; static inline struct hugetlb_cgroup *hugetlb_cgroup_from_page(struct page *page, bool reserved) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 325d5454bf168..8d8aa89a9928e 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -665,6 +665,16 @@ struct resv_map *resv_map_alloc(void) INIT_LIST_HEAD(&resv_map->regions); resv_map->adds_in_progress = 0; +#ifdef CONFIG_CGROUP_HUGETLB + /* + * Initialize these to 0. On shared mappings, 0's here indicate these + * fields don't do cgroup accounting. On private mappings, these will be + * re-initialized to the proper values, to indicate that hugetlb cgroup + * reservations are to be un-charged from here. + */ + resv_map->reservation_counter = NULL; + resv_map->pages_per_hpage = 0; +#endif INIT_LIST_HEAD(&resv_map->region_cache); list_add(&rg->link, &resv_map->region_cache); @@ -3216,7 +3226,18 @@ static void hugetlb_vm_op_close(struct vm_area_struct *vma) reserve = (end - start) - region_count(resv, start, end); - kref_put(&resv->refs, resv_map_release); +#ifdef CONFIG_CGROUP_HUGETLB + /* + * Since we check for HPAGE_RESV_OWNER above, this must a private + * mapping, and these values should be none-zero, and should point to + * the hugetlb_cgroup counter to uncharge for this reservation. + */ + WARN_ON(!resv->reservation_counter); + WARN_ON(!resv->pages_per_hpage); + + hugetlb_cgroup_uncharge_counter(resv->reservation_counter, + (end - start) * resv->pages_per_hpage); +#endif if (reserve) { /* @@ -3226,6 +3247,8 @@ static void hugetlb_vm_op_close(struct vm_area_struct *vma) gbl_reserve = hugepage_subpool_put_pages(spool, reserve); hugetlb_acct_memory(h, -gbl_reserve); } + + kref_put(&resv->refs, resv_map_release); } static int hugetlb_vm_op_split(struct vm_area_struct *vma, unsigned long addr) @@ -4559,6 +4582,7 @@ int hugetlb_reserve_pages(struct inode *inode, struct hstate *h = hstate_inode(inode); struct hugepage_subpool *spool = subpool_inode(inode); struct resv_map *resv_map; + struct hugetlb_cgroup *h_cg; long gbl_reserve; /* This should never happen */ @@ -4592,12 +4616,30 @@ int hugetlb_reserve_pages(struct inode *inode, chg = region_chg(resv_map, from, to); } else { + /* Private mapping. */ resv_map = resv_map_alloc(); if (!resv_map) return -ENOMEM; chg = to - from; + if (hugetlb_cgroup_charge_cgroup(hstate_index(h), + chg * pages_per_huge_page(h), + &h_cg, true)) { + kref_put(&resv_map->refs, resv_map_release); + return -ENOMEM; + } + +#ifdef CONFIG_CGROUP_HUGETLB + /* + * Since this branch handles private mappings, we attach the + * counter to uncharge for this reservation off resv_map. + */ + resv_map->reservation_counter = + &h_cg->reserved_hugepage[hstate_index(h)]; + resv_map->pages_per_hpage = pages_per_huge_page(h); +#endif + set_vma_resv_map(vma, resv_map); set_vma_resv_flags(vma, HPAGE_RESV_OWNER); } diff --git a/mm/hugetlb_cgroup.c b/mm/hugetlb_cgroup.c index ac1500205faf7..a0ca4024888e1 100644 --- a/mm/hugetlb_cgroup.c +++ b/mm/hugetlb_cgroup.c @@ -19,18 +19,6 @@ #include #include -struct hugetlb_cgroup { - struct cgroup_subsys_state css; - /* - * the counter to account for hugepages from hugetlb. - */ - struct page_counter hugepage[HUGE_MAX_HSTATE]; - /* - * the counter to account for hugepage reservations from hugetlb. - */ - struct page_counter reserved_hugepage[HUGE_MAX_HSTATE]; -}; - #define MEMFILE_PRIVATE(x, val) (((x) << 16) | (val)) #define MEMFILE_IDX(val) (((val) >> 16) & 0xffff) #define MEMFILE_ATTR(val) ((val) & 0xffff)