From patchwork Wed Jul 20 14:06:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zach O'Keefe X-Patchwork-Id: 12924028 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3402C43334 for ; Wed, 20 Jul 2022 14:06:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B4DB6B0074; Wed, 20 Jul 2022 10:06:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 17A138E0001; Wed, 20 Jul 2022 10:06:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F37046B0078; Wed, 20 Jul 2022 10:06:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E4A656B0074 for ; Wed, 20 Jul 2022 10:06:24 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C12C140601 for ; Wed, 20 Jul 2022 14:06:24 +0000 (UTC) X-FDA: 79707653088.08.17099E6 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 4E73D40067 for ; Wed, 20 Jul 2022 14:06:23 +0000 (UTC) Received: by mail-pl1-f201.google.com with SMTP id c18-20020a170903235200b0016c37f6d48cso10513359plh.19 for ; Wed, 20 Jul 2022 07:06:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=nvt0g38UPWQARVq2jIGTphYoZFkzkKWnTTwKEnIQhfQ=; b=kOfoDGHB1Q9BQsPHDaoPg/ATquyXpM9RonSrVrlCbITcTc52Ft5gfFhXSXVsKGtpT8 PhvYIniMH9RTaBk+gFr5ZlU2V0cu2Da5rBmxfSBpmP02wNvp/9j+KcFAu5np5Rt+i/Kd xYjSTLqlMYENAD1F1oFsJF6+xoXfLlg6UuEm39SQfb5lwoDhe9BW8yYDRHaZBgUBHvNd QWxM+NZKxTEhRO5sM5TXM1BMUri6U6YuRtfi8/Gtpzwg7XjNg3FC/H13Wxipk8+iRh0O XHcLbUy8sAMMw5fe/phEkIW90Z+SF7FeYMKlWrFpvUeFVL0ZZggSwwEfDwLwDDuhB6/i uImA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=nvt0g38UPWQARVq2jIGTphYoZFkzkKWnTTwKEnIQhfQ=; b=kWpr30ATvO1MAOREiW21rtH5ow8/tgbxYSJIZB4wXYCBUwiCulDMoeBYjCzt96JGT1 3C8pg3EDiiIXBESEOEjC7DqbKXsXpN4sxRZusSDkYxfsCJFpAcWZ6krB5LFJEcBWJVpp JNYuimCl7e+qSsWtVB/o3H3EOw7vNcLZ7NMEH5d18hXSXXI/CnnR0K6+XXEFUgzCYzhH EBFjluR8t/mNbCXZS8+iWU0YlI8Y7vLH0JoJ4azvVJFhGwMVTeNey/DecT4wzRWpm0FL eWUsHa2+nJVzxICuBpB3cfNyyJC+aSKzGtN2H/XcQpdQeHghL7zot0P2+tenkHgtT5Id i6wQ== X-Gm-Message-State: AJIora+cr1wDPFROzLoVI+BiTAtepQlnD0w2Qte04KaOddtiT0MLTNql FH0wLiyzKvNJqjpTYzMaMQPUQslZgdRQ X-Google-Smtp-Source: AGRyM1t2zdRRt9DeLnkDsuT/2qqMN51waS6QwunsePs8o1uAuGPXyJWhmN0DJLPTbGeYSCqe/mK6qeN6fKZD X-Received: from zokeefe3.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:1b6]) (user=zokeefe job=sendgmr) by 2002:a17:902:ab95:b0:16d:12a8:2af1 with SMTP id f21-20020a170902ab9500b0016d12a82af1mr7385488plr.164.1658325982447; Wed, 20 Jul 2022 07:06:22 -0700 (PDT) Date: Wed, 20 Jul 2022 07:06:01 -0700 In-Reply-To: <20220720140603.1958773-1-zokeefe@google.com> Message-Id: <20220720140603.1958773-3-zokeefe@google.com> Mime-Version: 1.0 References: <20220720140603.1958773-1-zokeefe@google.com> X-Mailer: git-send-email 2.37.0.170.g444d1eabd0-goog Subject: [PATCH mm-unstable 2/4] mm/khugepaged: consistently order cc->is_khugepaged and pte_* checks From: "Zach O'Keefe" To: Andrew Morton Cc: linux-mm@kvack.org, Hugh Dickins , Yang Shi , Miaohe Lin , "Zach O'Keefe" ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kOfoDGHB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 33gvYYgcKCHMqfbVVWVXffXcV.TfdcZelo-ddbmRTb.fiX@flex--zokeefe.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=33gvYYgcKCHMqfbVVWVXffXcV.TfdcZelo-ddbmRTb.fiX@flex--zokeefe.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658325983; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nvt0g38UPWQARVq2jIGTphYoZFkzkKWnTTwKEnIQhfQ=; b=ekBUW7EnL1JSdnr36lxPyVDwVnT8T7YBrn+1NK2PWaZ8n0XENrhWayTwmdzX+bqDxYsAU9 2mAP0keO0WfvO/MRqJFYfcw6CUrcdRCa7AJlTiHc8gz3NNTA+lClIhsaI15nfZXNosTkuD 6UdW4ZfzyfhQah70x5LNTvvJSQ7TnJs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658325983; a=rsa-sha256; cv=none; b=rk1oR9Sj3t1Hw4AmH+cCMxpvbJe6utBIyXNXus8h4PI60dHCWhEPn625mdYpq0I0S2GK8X jokTz9l7WBrNjjCmeSxjzJF030jh+BwpbwuqD2pDX22dGbINxtAvruYitFz81U1fFZaH/0 b4l61zZoTkWbpe5974zVh4us2gKAXuM= X-Rspamd-Queue-Id: 4E73D40067 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=kOfoDGHB; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf04.hostedemail.com: domain of 33gvYYgcKCHMqfbVVWVXffXcV.TfdcZelo-ddbmRTb.fiX@flex--zokeefe.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=33gvYYgcKCHMqfbVVWVXffXcV.TfdcZelo-ddbmRTb.fiX@flex--zokeefe.bounces.google.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: db8py9to9mtrs1ost74wmfrm7rzx518a X-HE-Tag: 1658325983-877853 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: cc->is_khugepaged is used to predicate the khugepaged-only behavior of enforcing khugepaged heuristics limited by the sysfs knobs khugepaged_max_ptes_[none|swap|shared]. In branches where khugepaged_max_ptes_* is checked, consistently check cc->is_khugepaged first. Also, local counters (for comparison vs khugepaged_max_ptes_* limits) were previously incremented in the comparison expression. Some of these counters (unmapped) are additionally used outside of khugepaged_max_ptes_* enforcement, and all counters are communicated in tracepoints. Move the correct accounting of these counters before branching statements to avoid future errors due to C's short-circuiting evaluation. Fixes: 9fab4752a181 ("mm/khugepaged: add flag to predicate khugepaged-only behavior") Link: https://lore.kernel.org/linux-mm/Ys2qJm6FaOQcxkha@google.com/ Signed-off-by: Zach O'Keefe Reviewed-by: Yang Shi Acked-by: David Rientjes --- mm/khugepaged.c | 49 +++++++++++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 20 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index ecd28bfeab60..290422577172 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -574,9 +574,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, pte_t pteval = *_pte; if (pte_none(pteval) || (pte_present(pteval) && is_zero_pfn(pte_pfn(pteval)))) { + ++none_or_zero; if (!userfaultfd_armed(vma) && - (++none_or_zero <= khugepaged_max_ptes_none || - !cc->is_khugepaged)) { + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { continue; } else { result = SCAN_EXCEED_NONE_PTE; @@ -596,11 +597,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, VM_BUG_ON_PAGE(!PageAnon(page), page); - if (cc->is_khugepaged && page_mapcount(page) > 1 && - ++shared > khugepaged_max_ptes_shared) { - result = SCAN_EXCEED_SHARED_PTE; - count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); - goto out; + if (page_mapcount(page) > 1) { + ++shared; + if (cc->is_khugepaged && + shared > khugepaged_max_ptes_shared) { + result = SCAN_EXCEED_SHARED_PTE; + count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); + goto out; + } } if (PageCompound(page)) { @@ -1170,8 +1174,9 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, _pte++, _address += PAGE_SIZE) { pte_t pteval = *_pte; if (is_swap_pte(pteval)) { - if (++unmapped <= khugepaged_max_ptes_swap || - !cc->is_khugepaged) { + ++unmapped; + if (!cc->is_khugepaged || + unmapped <= khugepaged_max_ptes_swap) { /* * Always be strict with uffd-wp * enabled swap entries. Please see @@ -1189,9 +1194,10 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, } } if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { + ++none_or_zero; if (!userfaultfd_armed(vma) && - (++none_or_zero <= khugepaged_max_ptes_none || - !cc->is_khugepaged)) { + (!cc->is_khugepaged || + none_or_zero <= khugepaged_max_ptes_none)) { continue; } else { result = SCAN_EXCEED_NONE_PTE; @@ -1221,12 +1227,14 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, goto out_unmap; } - if (cc->is_khugepaged && - page_mapcount(page) > 1 && - ++shared > khugepaged_max_ptes_shared) { - result = SCAN_EXCEED_SHARED_PTE; - count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); - goto out_unmap; + if (page_mapcount(page) > 1) { + ++shared; + if (cc->is_khugepaged && + shared > khugepaged_max_ptes_shared) { + result = SCAN_EXCEED_SHARED_PTE; + count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); + goto out_unmap; + } } page = compound_head(page); @@ -1961,8 +1969,9 @@ static int khugepaged_scan_file(struct mm_struct *mm, struct file *file, continue; if (xa_is_value(page)) { + ++swap; if (cc->is_khugepaged && - ++swap > khugepaged_max_ptes_swap) { + swap > khugepaged_max_ptes_swap) { result = SCAN_EXCEED_SWAP_PTE; count_vm_event(THP_SCAN_EXCEED_SWAP_PTE); break; @@ -2013,8 +2022,8 @@ static int khugepaged_scan_file(struct mm_struct *mm, struct file *file, rcu_read_unlock(); if (result == SCAN_SUCCEED) { - if (present < HPAGE_PMD_NR - khugepaged_max_ptes_none && - cc->is_khugepaged) { + if (cc->is_khugepaged && + present < HPAGE_PMD_NR - khugepaged_max_ptes_none) { result = SCAN_EXCEED_NONE_PTE; count_vm_event(THP_SCAN_EXCEED_NONE_PTE); } else {