From patchwork Tue Mar 8 21:34:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zach O'Keefe X-Patchwork-Id: 12774398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84292C433F5 for ; Tue, 8 Mar 2022 21:35:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A04D58D000C; Tue, 8 Mar 2022 16:35:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 919188D0001; Tue, 8 Mar 2022 16:35:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 792D48D000C; Tue, 8 Mar 2022 16:35:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id 6B5168D0001 for ; Tue, 8 Mar 2022 16:35:05 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4A0F122092 for ; Tue, 8 Mar 2022 21:35:05 +0000 (UTC) X-FDA: 79222524570.12.CEACCE5 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf30.hostedemail.com (Postfix) with ESMTP id D024E8000A for ; Tue, 8 Mar 2022 21:35:04 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id o41-20020a17090a0a2c00b001bf06e5badfso292173pjo.3 for ; Tue, 08 Mar 2022 13:35:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ipw1R7b8Sbc0tCBBdZYpbSNx+pW+oifi9JV5Pg9mDo0=; b=fxS7bc9XBrs7XecMQfIQjSg7jOIBkH3yMKI92zX0g4pPwRdVFxFGCuTg65X8qdiwqL F5WKEcMx6OZosejrxbuSVYpIRy5pvCkNwvGqeY7BXsYOnyZ7QD1a7i99Fiw7cNE2gxVB hmnjJOzym4cmlUskERwkaM4ZFI9LsXT71k/Saq3L8sqx1IyO5LKZPH9Ds3Yrx9SE8BY+ T8I+HHcq7h+r28z9DHOWWYmjMmiQRrA1W6FJGmLeKJOdkMcsSsfOVi/FkTldWUT1iKY8 xp1xlgI+wn6ytSAMygRXslP+g2wDUfnLcIYH/D3/LO+JBjk17slT4ArTH9rtGEaMvyL9 lIMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ipw1R7b8Sbc0tCBBdZYpbSNx+pW+oifi9JV5Pg9mDo0=; b=3M/d/UUmFd6tM1aqLi8+bDWcZqu6PA7BXTs/EUQ3p+RgxEA1kxTbo3lLsJI2k8h3Eh OtMfJio0icvMqv4p0X/FFbrxOI17dUsO0yrWknrroe+1ciK9TVo+5Q2qU12XykFT8pmK CX+yM6sHbdhEY+fzm+4cnZrbdm+0chlgkuRjKY0dFpcb62gcpBc4xfmuXDw0sAdrZze3 DGGVRsapZ/+vIInTH0xIie9JBifg+lTHrQQgns5zzifv7eMrIp5aJZMdAMAe09YKFCuH NNwJIIwjENJJURCCGhddRRVvEJ7Y7omRhOwrMkqjF4iTwYR2t0UpQlsS5rErQktxl5D/ 1bag== X-Gm-Message-State: AOAM533Sd7kK8L6qPh7nkTAFQPMk3e/YKc9z6OmMPJCyh36AFUAB99UU /5OsEk0+cW0kZUxDVodRRHIAacTJjRay X-Google-Smtp-Source: ABdhPJwUx1r259FM6SlG9TMDWvRlaqM9dHNFlQFgyb8wj6p+Zp8zdYXbB/9toXLIq1WtsUepwejsFU/BMoDx X-Received: from zokeefe3.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1b6]) (user=zokeefe job=sendgmr) by 2002:a05:6a00:244c:b0:4f6:67b8:a6b4 with SMTP id d12-20020a056a00244c00b004f667b8a6b4mr20403199pfj.51.1646775303868; Tue, 08 Mar 2022 13:35:03 -0800 (PST) Date: Tue, 8 Mar 2022 13:34:12 -0800 In-Reply-To: <20220308213417.1407042-1-zokeefe@google.com> Message-Id: <20220308213417.1407042-10-zokeefe@google.com> Mime-Version: 1.0 References: <20220308213417.1407042-1-zokeefe@google.com> X-Mailer: git-send-email 2.35.1.616.g0bdcbb4464-goog Subject: [RFC PATCH 09/14] mm/khugepaged: record SCAN_PAGE_COMPOUND when scan_pmd() finds THP From: "Zach O'Keefe" To: Alex Shi , David Hildenbrand , David Rientjes , Michal Hocko , Pasha Tatashin , SeongJae Park , Song Liu , Vlastimil Babka , Zi Yan , linux-mm@kvack.org Cc: Andrea Arcangeli , Andrew Morton , Arnd Bergmann , Axel Rasmussen , Chris Kennelly , Chris Zankel , Helge Deller , Hugh Dickins , Ivan Kokshaysky , "James E.J. Bottomley" , Jens Axboe , "Kirill A. Shutemov" , Matthew Wilcox , Matt Turner , Max Filippov , Miaohe Lin , Minchan Kim , Patrick Xia , Pavel Begunkov , Peter Xu , Richard Henderson , Thomas Bogendoerfer , Yang Shi , "Zach O'Keefe" X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D024E8000A X-Stat-Signature: qoqnxmc1hfwrcgrzdzq6ybm7iipxiyz3 Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=fxS7bc9X; spf=pass (imf30.hostedemail.com: domain of 3B8wnYgcKCFYNC822324CC492.0CA96BIL-AA8Jy08.CF4@flex--zokeefe.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3B8wnYgcKCFYNC822324CC492.0CA96BIL-AA8Jy08.CF4@flex--zokeefe.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-HE-Tag: 1646775304-943476 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When scanning an anon pmd to see if it's eligible for collapse, return SCAN_PAGE_COMPOUND if the pmd already maps a thp. This is consistent with handling when scanning file-backed memory. Signed-off-by: Zach O'Keefe --- mm/khugepaged.c | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index ecbd3fc41c80..403578161a3b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,6 +1011,38 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, vm_flags_ignore, vmap); } +/* + * If returning NULL (meaning the pmd isn't mapped, isn't present, or thp), + * write the reason to *result. + */ +static pmd_t *find_pmd_or_thp_or_none(struct mm_struct *mm, + unsigned long address, + int *result) +{ + pmd_t *pmd = mm_find_pmd_raw(mm, address); + pmd_t pmde; + + if (!pmd) { + *result = SCAN_PMD_NULL; + return NULL; + } + + pmde = pmd_read_atomic(pmd); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + /* See comments in pmd_none_or_trans_huge_or_clear_bad() */ + barrier(); +#endif + if (!pmd_present(pmde) || !pmd_none(pmde)) { + *result = SCAN_PMD_NULL; + return NULL; + } else if (pmd_trans_huge(pmde)) { + *result = SCAN_PAGE_COMPOUND; + return NULL; + } + return pmd; +} + /* * Bring missing pages in from swap, to complete THP collapse. * Only done if khugepaged_scan_pmd believes it is worthwhile. @@ -1212,9 +1244,8 @@ static void collapse_huge_page(struct mm_struct *mm, goto out_nolock; } - pmd = mm_find_pmd(mm, address); + pmd = find_pmd_or_thp_or_none(mm, address, &result); if (!pmd) { - result = SCAN_PMD_NULL; mmap_read_unlock(mm); goto out_nolock; } @@ -1287,11 +1318,9 @@ static void scan_pmd(struct mm_struct *mm, mmap_assert_locked(mm); VM_BUG_ON(address & ~HPAGE_PMD_MASK); - pmd = mm_find_pmd(mm, address); - if (!pmd) { - scan_result->result = SCAN_PMD_NULL; + pmd = find_pmd_or_thp_or_none(mm, address, &scan_result->result); + if (!pmd) goto out; - } memset(cc->node_load, 0, sizeof(cc->node_load)); pte = pte_offset_map_lock(mm, pmd, address, &ptl);