From patchwork Tue Jun 5 13:29:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 10448275 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 8BA5360467 for ; Tue, 5 Jun 2018 13:32:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 722A729387 for ; Tue, 5 Jun 2018 13:32:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6EF252948F; Tue, 5 Jun 2018 13:32:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DC395294E1 for ; Tue, 5 Jun 2018 13:32:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752187AbeFENcN (ORCPT ); Tue, 5 Jun 2018 09:32:13 -0400 Received: from mail-qt0-f175.google.com ([209.85.216.175]:34792 "EHLO mail-qt0-f175.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752024AbeFENaB (ORCPT ); Tue, 5 Jun 2018 09:30:01 -0400 Received: by mail-qt0-f175.google.com with SMTP id d3-v6so2369846qto.1 for ; Tue, 05 Jun 2018 06:30:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id:in-reply-to:references; bh=88IO6KZPW8Znnx5GcdehjkvLsaYRLfZayiNTas2ZDDQ=; b=fmKH//qFl2SfSEQ4SABIgeLIlksk3pyfpbj3OzjMLBd679WdiSf22bkhqVQW7y7SDz 98Q4sRyiSYgc07RsSR2tzUe9NTyWLp2w7mrBUjHHC3ShnMqUO9zLcAbGbRQ5iLlpSJRu BuFWgIUg1ZG1YoXuQAGa88dnHzl1JAoJxpKFwAlB7uDs/FjYmd9eDb2qmX83m9aPvdKT 3/+4/NVmDkV4Eud82B2pfIXvo2lpHyrnVl6EL/ZLkK5u4/rxweb5MlljUF6rBG1XK8V9 69TiZ64UbAn6kXB4LbgN/aTJHMD/iwrd4Xy8b3Wd3b2f3D9ZJftoXVZgXH7w25EkbMdv Z52w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=88IO6KZPW8Znnx5GcdehjkvLsaYRLfZayiNTas2ZDDQ=; b=CVS/sJe/yVZ1APMEvSZCWG8V53RoydAC9PpWhs7ScsBZzY6ESK9B7Idh2B5RJja3HQ anx4cNymXDt2dCzsXzo/4VgYUxnKUfM+dJaqBFALZMjlte6QjU2F9iG32aaOGf8voD1L 8d9gxbOwwKq/kVgdbDvh+4WHY9VR355Zs/n+Xes22xPo7rHhrHlDQ7akmqBHqrALBfpQ PSLibzVwIrkmWdtpvPwCq0gWukU6c+h6Bf39HDNB1HTuz40h12RHGmDNNk94W23G6Xz7 4ruaYTftgrAlZe2TKI3BjpcR9uY6YffuhxLSfr4tRiEwECwwJO1GpOhFnuJm+n2qKcIv Hefw== X-Gm-Message-State: APt69E2Z+Z06iJ106Xcx15WYFXOOUASgsCnMsI7m9/kYI34hUOnLwkD5 W9QBuHf3k8/mqJdSWUSfupWTIw== X-Google-Smtp-Source: ADUXVKIuH9McsQv6az96PPMKbNaOEiwC+h/AtuJhu0Tq3xRZZtCWNCjCK2Aq7to3BXzTW2BAVSP1Zg== X-Received: by 2002:aed:3942:: with SMTP id l60-v6mr8696864qte.67.1528205400920; Tue, 05 Jun 2018 06:30:00 -0700 (PDT) Received: from localhost ([107.15.81.208]) by smtp.gmail.com with ESMTPSA id a18-v6sm25472482qkj.60.2018.06.05.06.30.00 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 05 Jun 2018 06:30:00 -0700 (PDT) From: Josef Bacik To: axboe@kernel.dk, kernel-team@fb.com, linux-block@vger.kernel.org, akpm@linux-foundation.org, hannes@cmpxchg.org, linux-kernel@vger.kernel.org, tj@kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH 07/13] memcontrol: schedule throttling if we are congested Date: Tue, 5 Jun 2018 09:29:42 -0400 Message-Id: <20180605132948.1664-8-josef@toxicpanda.com> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180605132948.1664-1-josef@toxicpanda.com> References: <20180605132948.1664-1-josef@toxicpanda.com> Sender: linux-block-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Tejun Heo Memory allocations can induce swapping via kswapd or direct reclaim. If we are having IO done for us by kswapd and don't actually go into direct reclaim we may never get scheduled for throttling. So instead check to see if our cgroup is congested, and if so schedule the throttling. Before we return to user space the throttling stuff will only throttle if we actually required it. Signed-off-by: Tejun Heo Acked-by: Tejun Heo Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 13 +++++++++++++ include/linux/swap.h | 8 ++++++++ mm/huge_memory.c | 6 +++--- mm/memcontrol.c | 13 +++++++++++++ mm/memory.c | 11 ++++++----- mm/shmem.c | 10 +++++----- mm/swapfile.c | 24 ++++++++++++++++++++++++ 7 files changed, 72 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index d99b71bc2c66..4d2e7f35f2dc 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -290,6 +290,9 @@ bool mem_cgroup_low(struct mem_cgroup *root, struct mem_cgroup *memcg); int mem_cgroup_try_charge(struct page *page, struct mm_struct *mm, gfp_t gfp_mask, struct mem_cgroup **memcgp, bool compound); +int mem_cgroup_try_charge_delay(struct page *page, struct mm_struct *mm, + gfp_t gfp_mask, struct mem_cgroup **memcgp, + bool compound); void mem_cgroup_commit_charge(struct page *page, struct mem_cgroup *memcg, bool lrucare, bool compound); void mem_cgroup_cancel_charge(struct page *page, struct mem_cgroup *memcg, @@ -745,6 +748,16 @@ static inline int mem_cgroup_try_charge(struct page *page, struct mm_struct *mm, return 0; } +static inline int mem_cgroup_try_charge_delay(struct page *page, + struct mm_struct *mm, + gfp_t gfp_mask, + struct mem_cgroup **memcgp, + bool compound) +{ + *memcgp = NULL; + return 0; +} + static inline void mem_cgroup_commit_charge(struct page *page, struct mem_cgroup *memcg, bool lrucare, bool compound) diff --git a/include/linux/swap.h b/include/linux/swap.h index 2417d288e016..7ba0f52496e0 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -630,11 +630,19 @@ static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) return memcg->swappiness; } +extern void mem_cgroup_throttle_swaprate(struct mem_cgroup *memcg, int node, + gfp_t gfp_mask); #else static inline int mem_cgroup_swappiness(struct mem_cgroup *mem) { return vm_swappiness; } + +static inline void mem_cgroup_throttle_swaprate(struct mem_cgroup *memcg, + int node, gfp_t gfP_maks) +{ +} + #endif #ifdef CONFIG_MEMCG_SWAP diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a3a1815f8e11..9812ddad9961 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -555,7 +555,7 @@ static int __do_huge_pmd_anonymous_page(struct vm_fault *vmf, struct page *page, VM_BUG_ON_PAGE(!PageCompound(page), page); - if (mem_cgroup_try_charge(page, vma->vm_mm, gfp, &memcg, true)) { + if (mem_cgroup_try_charge_delay(page, vma->vm_mm, gfp, &memcg, true)) { put_page(page); count_vm_event(THP_FAULT_FALLBACK); return VM_FAULT_FALLBACK; @@ -1145,7 +1145,7 @@ static int do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pmd_t orig_pmd, pages[i] = alloc_page_vma_node(GFP_HIGHUSER_MOVABLE, vma, vmf->address, page_to_nid(page)); if (unlikely(!pages[i] || - mem_cgroup_try_charge(pages[i], vma->vm_mm, + mem_cgroup_try_charge_delay(pages[i], vma->vm_mm, GFP_KERNEL, &memcg, false))) { if (pages[i]) put_page(pages[i]); @@ -1315,7 +1315,7 @@ int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd) goto out; } - if (unlikely(mem_cgroup_try_charge(new_page, vma->vm_mm, + if (unlikely(mem_cgroup_try_charge_delay(new_page, vma->vm_mm, huge_gfp, &memcg, true))) { put_page(new_page); split_huge_pmd(vma, vmf->pmd, vmf->address); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2bd3df3d101a..5fffd28477c7 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5458,6 +5458,19 @@ int mem_cgroup_try_charge(struct page *page, struct mm_struct *mm, return ret; } +int mem_cgroup_try_charge_delay(struct page *page, struct mm_struct *mm, + gfp_t gfp_mask, struct mem_cgroup **memcgp, + bool compound) +{ + struct mem_cgroup *memcg; + int ret; + + ret = mem_cgroup_try_charge(page, mm, gfp_mask, memcgp, compound); + memcg = *memcgp; + mem_cgroup_throttle_swaprate(memcg, page_to_nid(page), gfp_mask); + return ret; +} + /** * mem_cgroup_commit_charge - commit a page charge * @page: page to charge diff --git a/mm/memory.c b/mm/memory.c index 01f5464e0fd2..d0eea6d33b18 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2494,7 +2494,7 @@ static int wp_page_copy(struct vm_fault *vmf) cow_user_page(new_page, old_page, vmf->address, vma); } - if (mem_cgroup_try_charge(new_page, mm, GFP_KERNEL, &memcg, false)) + if (mem_cgroup_try_charge_delay(new_page, mm, GFP_KERNEL, &memcg, false)) goto oom_free_new; __SetPageUptodate(new_page); @@ -2994,8 +2994,8 @@ int do_swap_page(struct vm_fault *vmf) goto out_page; } - if (mem_cgroup_try_charge(page, vma->vm_mm, GFP_KERNEL, - &memcg, false)) { + if (mem_cgroup_try_charge_delay(page, vma->vm_mm, GFP_KERNEL, + &memcg, false)) { ret = VM_FAULT_OOM; goto out_page; } @@ -3156,7 +3156,8 @@ static int do_anonymous_page(struct vm_fault *vmf) if (!page) goto oom; - if (mem_cgroup_try_charge(page, vma->vm_mm, GFP_KERNEL, &memcg, false)) + if (mem_cgroup_try_charge_delay(page, vma->vm_mm, GFP_KERNEL, &memcg, + false)) goto oom_free_page; /* @@ -3652,7 +3653,7 @@ static int do_cow_fault(struct vm_fault *vmf) if (!vmf->cow_page) return VM_FAULT_OOM; - if (mem_cgroup_try_charge(vmf->cow_page, vma->vm_mm, GFP_KERNEL, + if (mem_cgroup_try_charge_delay(vmf->cow_page, vma->vm_mm, GFP_KERNEL, &vmf->memcg, false)) { put_page(vmf->cow_page); return VM_FAULT_OOM; diff --git a/mm/shmem.c b/mm/shmem.c index 9d6c7e595415..a96af5690864 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1219,8 +1219,8 @@ int shmem_unuse(swp_entry_t swap, struct page *page) * the shmem_swaplist_mutex which might hold up shmem_writepage(). * Charged back to the user (not to caller) when swap account is used. */ - error = mem_cgroup_try_charge(page, current->mm, GFP_KERNEL, &memcg, - false); + error = mem_cgroup_try_charge_delay(page, current->mm, GFP_KERNEL, + &memcg, false); if (error) goto out; /* No radix_tree_preload: swap entry keeps a place for page in tree */ @@ -1697,7 +1697,7 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index, goto failed; } - error = mem_cgroup_try_charge(page, charge_mm, gfp, &memcg, + error = mem_cgroup_try_charge_delay(page, charge_mm, gfp, &memcg, false); if (!error) { error = shmem_add_to_page_cache(page, mapping, index, @@ -1803,7 +1803,7 @@ alloc_nohuge: page = shmem_alloc_and_acct_page(gfp, inode, if (sgp == SGP_WRITE) __SetPageReferenced(page); - error = mem_cgroup_try_charge(page, charge_mm, gfp, &memcg, + error = mem_cgroup_try_charge_delay(page, charge_mm, gfp, &memcg, PageTransHuge(page)); if (error) goto unacct; @@ -2276,7 +2276,7 @@ static int shmem_mfill_atomic_pte(struct mm_struct *dst_mm, __SetPageSwapBacked(page); __SetPageUptodate(page); - ret = mem_cgroup_try_charge(page, dst_mm, gfp, &memcg, false); + ret = mem_cgroup_try_charge_delay(page, dst_mm, gfp, &memcg, false); if (ret) goto out_release; diff --git a/mm/swapfile.c b/mm/swapfile.c index cc2cf04d9018..a42f86bc55ad 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3725,6 +3725,30 @@ static void free_swap_count_continuations(struct swap_info_struct *si) } } +#ifdef CONFIG_MEMCG +void mem_cgroup_throttle_swaprate(struct mem_cgroup *memcg, int node, + gfp_t gfp_mask) +{ + struct swap_info_struct *si, *next; + if (!(gfp_mask & __GFP_IO) || !memcg) + return; + + if (atomic_read(&memcg->css.cgroup->congestion_count) == 0) + return; + + spin_lock(&swap_avail_lock); + plist_for_each_entry_safe(si, next, &swap_avail_heads[node], + avail_lists[node]) { + if (si->bdev) { + blkcg_schedule_throttle(bdev_get_queue(si->bdev), + true); + break; + } + } + spin_unlock(&swap_avail_lock); +} +#endif + static int __init swapfile_init(void) { int nid;