From patchwork Sat Jun 4 00:39:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zach O'Keefe X-Patchwork-Id: 12869458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93D40C43334 for ; Sat, 4 Jun 2022 00:40:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C22C88D0003; Fri, 3 Jun 2022 20:40:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BD0108D0001; Fri, 3 Jun 2022 20:40:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4A448D0003; Fri, 3 Jun 2022 20:40:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 8C3D48D0001 for ; Fri, 3 Jun 2022 20:40:15 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6568A35FFC for ; Sat, 4 Jun 2022 00:40:15 +0000 (UTC) X-FDA: 79538696790.29.02561DB Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf28.hostedemail.com (Postfix) with ESMTP id D3C26C0037 for ; Sat, 4 Jun 2022 00:39:33 +0000 (UTC) Received: by mail-pj1-f74.google.com with SMTP id j15-20020a17090a738f00b001e345e429d2so4874961pjg.0 for ; Fri, 03 Jun 2022 17:40:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=etDzY7WRCSKD998NSJzC9UFp2r1Hcs3dHIWpMpnYzaQ=; b=rtdpM5FP01DnZvGKeE4PuwH9Xm2JZTNdWAOLKL7n+mWwg5ppXPHEwerIqxsM4uCDlK KOdLeHToYLVEsPw6Z7ycl6jVZuNacxqxMoKSZwY12ZZQXnv+V488KNAWDl4o2tbnifDU DRVdYjWaO/vvkWuuMSC0GNe4+xnPl0CgAeTnoJkYqDAaHTN3mGC+4F2gbuMABZd6vSjZ ZHQAnoSHEDZgONv+OfSSzn57HKtqI3wz94EGonps3+kfiWQCLDdT1wm8k8OkZO4sXy1T TZA7V6c7eo66ZVmrv8nTq5exYCedDmpFyxrsQYqkpkHy4KHsiYBqkwJeoy5Nfgeb4dxv IMjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=etDzY7WRCSKD998NSJzC9UFp2r1Hcs3dHIWpMpnYzaQ=; b=plpxN6Pts5FNYnq6I0zJFGsIKS7lpME0NGB3a2tTWV/31xdB/GziS5o7wYswJgACTa D24RDVPbrH5BDSLqiss41DF7h7nY7ttN8ddDQcu0BqcMXQ/y/xVxU3BSeUseAeclIZtz PcPZbcfpnoxl07lEXaC40aLV1AZLyCEdawS/ZXlCRMGywipkwfL5qH1QJoEQNa5eBObN ZK+Dgl/EcF300gs0+8XPuuv6RjRx80lmIP+NrYmpDLH4QbVspo4AkuqeMN/G+K6G0Vhg 02SHjt0sy96gOp0COnS8JoR0EpQQN0Re9T7kgbUhzKSLAEY/dIIJvxdrEe9fUYz8ZUXZ JAFQ== X-Gm-Message-State: AOAM5313KOLirEVF5LwKDXu3Vx4eh1cwjm+OmFn+c1jI9DKt04kUfFN4 fw+CS5jeX4b4C5R3PDHeJfbCu/UU66Ap X-Google-Smtp-Source: ABdhPJxUZw/lmb0vuJ1f6s7zHXiNCAeKxG/hCRGfcN/qm3IQDEsuDjA/jeez/d5OaCK4kn13wT6p2cXs0bja X-Received: from zokeefe3.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1b6]) (user=zokeefe job=sendgmr) by 2002:a17:903:186:b0:167:62b0:a556 with SMTP id z6-20020a170903018600b0016762b0a556mr211122plg.122.1654303213973; Fri, 03 Jun 2022 17:40:13 -0700 (PDT) Date: Fri, 3 Jun 2022 17:39:50 -0700 In-Reply-To: <20220604004004.954674-1-zokeefe@google.com> Message-Id: <20220604004004.954674-2-zokeefe@google.com> Mime-Version: 1.0 References: <20220604004004.954674-1-zokeefe@google.com> X-Mailer: git-send-email 2.36.1.255.ge46751e96f-goog Subject: [PATCH v6 01/15] mm: khugepaged: don't carry huge page to the next loop for !CONFIG_NUMA From: "Zach O'Keefe" To: Alex Shi , David Hildenbrand , David Rientjes , Matthew Wilcox , Michal Hocko , Pasha Tatashin , Peter Xu , Rongwei Wang , SeongJae Park , Song Liu , Vlastimil Babka , Yang Shi , Zi Yan , linux-mm@kvack.org Cc: Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Thomas Bogendoerfer , "Zach O'Keefe" Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=rtdpM5FP; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf28.hostedemail.com: domain of 37amaYgcKCMYB0wqqrqs00sxq.o0yxuz69-yyw7mow.03s@flex--zokeefe.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=37amaYgcKCMYB0wqqrqs00sxq.o0yxuz69-yyw7mow.03s@flex--zokeefe.bounces.google.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D3C26C0037 X-Rspam-User: X-Stat-Signature: nunfo11f4xgkwdjfcfkf34psk5i9ddsx X-HE-Tag: 1654303173-46532 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Yang Shi The khugepaged has optimization to reduce huge page allocation calls for !CONFIG_NUMA by carrying the allocated but failed to collapse huge page to the next loop. CONFIG_NUMA doesn't do so since the next loop may try to collapse huge page from a different node, so it doesn't make too much sense to carry it. But when NUMA=n, the huge page is allocated by khugepaged_prealloc_page() before scanning the address space, so it means huge page may be allocated even though there is no suitable range for collapsing. Then the page would be just freed if khugepaged already made enough progress. This could make NUMA=n run have 5 times as much thp_collapse_alloc as NUMA=y run. This problem actually makes things worse due to the way more pointless THP allocations and makes the optimization pointless. This could be fixed by carrying the huge page across scans, but it will complicate the code further and the huge page may be carried indefinitely. But if we take one step back, the optimization itself seems not worth keeping nowadays since: * Not too many users build NUMA=n kernel nowadays even though the kernel is actually running on a non-NUMA machine. Some small devices may run NUMA=n kernel, but I don't think they actually use THP. * Since commit 44042b449872 ("mm/page_alloc: allow high-order pages to be stored on the per-cpu lists"), THP could be cached by pcp. This actually somehow does the job done by the optimization. Cc: Hugh Dickins Cc: "Kirill A. Shutemov" Signed-off-by: Zach O'Keefe --- mm/khugepaged.c | 100 ++++++++---------------------------------------- 1 file changed, 17 insertions(+), 83 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 476d79360101..cc3d6fb446d5 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -833,29 +833,30 @@ static int khugepaged_find_target_node(void) last_khugepaged_target_node = target_node; return target_node; } +#else +static int khugepaged_find_target_node(void) +{ + return 0; +} +#endif -static bool khugepaged_prealloc_page(struct page **hpage, bool *wait) +/* Sleep for the first alloc fail, break the loop for the second fail */ +static bool alloc_fail_should_sleep(struct page **hpage, bool *wait) { if (IS_ERR(*hpage)) { if (!*wait) - return false; + return true; *wait = false; *hpage = NULL; khugepaged_alloc_sleep(); - } else if (*hpage) { - put_page(*hpage); - *hpage = NULL; } - - return true; + return false; } static struct page * khugepaged_alloc_page(struct page **hpage, gfp_t gfp, int node) { - VM_BUG_ON_PAGE(*hpage, *hpage); - *hpage = __alloc_pages_node(node, gfp, HPAGE_PMD_ORDER); if (unlikely(!*hpage)) { count_vm_event(THP_COLLAPSE_ALLOC_FAILED); @@ -867,74 +868,6 @@ khugepaged_alloc_page(struct page **hpage, gfp_t gfp, int node) count_vm_event(THP_COLLAPSE_ALLOC); return *hpage; } -#else -static int khugepaged_find_target_node(void) -{ - return 0; -} - -static inline struct page *alloc_khugepaged_hugepage(void) -{ - struct page *page; - - page = alloc_pages(alloc_hugepage_khugepaged_gfpmask(), - HPAGE_PMD_ORDER); - if (page) - prep_transhuge_page(page); - return page; -} - -static struct page *khugepaged_alloc_hugepage(bool *wait) -{ - struct page *hpage; - - do { - hpage = alloc_khugepaged_hugepage(); - if (!hpage) { - count_vm_event(THP_COLLAPSE_ALLOC_FAILED); - if (!*wait) - return NULL; - - *wait = false; - khugepaged_alloc_sleep(); - } else - count_vm_event(THP_COLLAPSE_ALLOC); - } while (unlikely(!hpage) && likely(khugepaged_enabled())); - - return hpage; -} - -static bool khugepaged_prealloc_page(struct page **hpage, bool *wait) -{ - /* - * If the hpage allocated earlier was briefly exposed in page cache - * before collapse_file() failed, it is possible that racing lookups - * have not yet completed, and would then be unpleasantly surprised by - * finding the hpage reused for the same mapping at a different offset. - * Just release the previous allocation if there is any danger of that. - */ - if (*hpage && page_count(*hpage) > 1) { - put_page(*hpage); - *hpage = NULL; - } - - if (!*hpage) - *hpage = khugepaged_alloc_hugepage(wait); - - if (unlikely(!*hpage)) - return false; - - return true; -} - -static struct page * -khugepaged_alloc_page(struct page **hpage, gfp_t gfp, int node) -{ - VM_BUG_ON(!*hpage); - - return *hpage; -} -#endif /* * If mmap_lock temporarily dropped, revalidate vma @@ -1188,8 +1121,10 @@ static void collapse_huge_page(struct mm_struct *mm, out_up_write: mmap_write_unlock(mm); out_nolock: - if (!IS_ERR_OR_NULL(*hpage)) + if (!IS_ERR_OR_NULL(*hpage)) { mem_cgroup_uncharge(page_folio(*hpage)); + put_page(*hpage); + } trace_mm_collapse_huge_page(mm, isolated, result); return; } @@ -1992,8 +1927,10 @@ static void collapse_file(struct mm_struct *mm, unlock_page(new_page); out: VM_BUG_ON(!list_empty(&pagelist)); - if (!IS_ERR_OR_NULL(*hpage)) + if (!IS_ERR_OR_NULL(*hpage)) { mem_cgroup_uncharge(page_folio(*hpage)); + put_page(*hpage); + } /* TODO: tracepoints */ } @@ -2243,7 +2180,7 @@ static void khugepaged_do_scan(void) lru_add_drain_all(); while (progress < pages) { - if (!khugepaged_prealloc_page(&hpage, &wait)) + if (alloc_fail_should_sleep(&hpage, &wait)) break; cond_resched(); @@ -2262,9 +2199,6 @@ static void khugepaged_do_scan(void) progress = pages; spin_unlock(&khugepaged_mm_lock); } - - if (!IS_ERR_OR_NULL(hpage)) - put_page(hpage); } static bool khugepaged_should_wakeup(void)