From patchwork Mon Dec 16 16:51:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BE79E77180 for ; Mon, 16 Dec 2024 16:52:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 13D0C6B00C7; Mon, 16 Dec 2024 11:52:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C6BC6B00C8; Mon, 16 Dec 2024 11:52:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA9946B00C9; Mon, 16 Dec 2024 11:52:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CACFC6B00C7 for ; Mon, 16 Dec 2024 11:52:55 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 73C38A0CD1 for ; Mon, 16 Dec 2024 16:52:55 +0000 (UTC) X-FDA: 82901416080.18.7E1194B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id D274AC0013 for ; Mon, 16 Dec 2024 16:52:22 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367953; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ThaVIYRuc3vYNUyLNOH6vkpdMl8i5Jto84xNiz/AIKE=; b=Bmvxki87mo39Jg5M6srb7rZxMd9VQY+k/a+IIpatwACBZcSru6HfX824sbxNlNN+DLrcw8 LyLMgg7SjDSIuTj4pBEGEUjr+ddrWBbYe7g7PxcSLfOa1Iernu0FiGvTzS2pvVjkpwEE7U 1jIY+qfA+QoYLRdmZhe0s0iOGB1l1Kg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367953; a=rsa-sha256; cv=none; b=CW9uIqByQ1GAEmq5PYRc2je4weDVyD+ghJcGfaC+CCYIGxvE5HtkHta+p2MmiJjmzha0x6 t0B2cJGevg2cZqRK0HLZGzCAdAq8YHkpxzBsghcGqPPdMkdimGodUwRsSmK1LDTuhknmJx Rq+a++yKtJOXAJQc07JzrcIwnwXa+Ss= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 42BAF16F8; Mon, 16 Dec 2024 08:53:21 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A83103F528; Mon, 16 Dec 2024 08:52:42 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 07/12] khugepaged: Scan PTEs order-wise Date: Mon, 16 Dec 2024 22:21:00 +0530 Message-Id: <20241216165105.56185-8-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: f4frxzgjc8x3rpxdgbnjmo3te5dfi5d9 X-Rspamd-Queue-Id: D274AC0013 X-Rspam-User: X-HE-Tag: 1734367942-999431 X-HE-Meta: U2FsdGVkX19XAo5Of1Fz9Dg4QpHRItZy/A5UqyKTsRtYg0QApzP8gQArr7pUeoVVLJUTZrn4cqx320rskMkK5KOuFj5UBTjxtNl81B3Qt7WPjZ9VVI/rPk12/01oRZf/VNb3IbK3fy1oY1nViDMn5/7fYvzcBWmoixfX4n5gv3pJxC+vnLrVaGbeWUTuikcraaYZjuAnmTv+/JLRQ7/QmoWjjEMgsG1ANrGLhB1Iqx0aR8nAOsEoXMab6bpEwU5l/w7j0gSVNrO5X8sAVgqdJJqVhN5L1qlMVVWrB6VEY+Zv5+RDJgKL5gPxUJq87ERm2nGgqKvI4hEE9iIJRZfqWFlG6RzdvpPkmV1rOmUlaziXbb30aJbZ7VAZXIndLlYJ9KcY4kQzi32oBA1kUhmNuOrYzTG9qcDEgX2t3AunUSQrFzc9RDRxLstW1kOKUmoGgSD6MHdmDX4rZVndW1SdPGv/EwOuc09fs+ccZ9mZTVVVyfFC1EkiqvwtNS9X2//MQPjjBCGg+nprP8gAidmG7L2NrP6fMAucXvojpN4MT+6QK3Vk6HiUkTAv5GbOVs4MHzJyPqU8d+uTxLTZsE+b9L6PWPG0y+5aYv2RC8MfHPxWDkd0784u6M8/ycN9Vn+8ZFdy/b1w1TJOpUvvzv6ZoUbhBcSUdug+PsBzJJAvypzzYcdElUGZOA/fBVQvOyCCjr5FJsgGes3g4uBIlHLy2ph49Gqxv4BO17mWJ7bIC87P27TLsc3sfjtfV2mscgNAKAnh+ZMw6fyc7KZ5xAiHD8d1gEtnCXovWrXFV7C6ZzMSh5l4LPU4gtMF425TXeh8mfVlLZm3og4cycPf7KMv2b3tD5K7uNx+Z0IdwOFjw/r/UXB0oXy7W0TK2NaiuquK/CXcAe3w5ddaitY3QExSzcD7IbRqTGw2ePo3bL6uOAe7ne7h8i8UBCv0xoDCSCNBFii5EngWM2mmtGLPf47 5NsjIndT vzD3bw+3gyJeU29TgSR6q7ivvwGROW9EceytuvMM5HFIzSzwuXCwzIo5ZzXs9LBV9cVifT4yuMpQJ7cUUD5DpiyAooS3hVyXKp67PI2lM1LSNFhQRGpQds2gyW5P5I/tggoWRhPsUGZ+6NeGJE/t9pt6Kz0lCa2MBqaNtux0Y8aO3eWSyqa0eYiO6XedPWHX4sSiqkgfWKh/pDa48xhdE1Fvu2iCg8XLnj3azKuZ7mN1ood3IWn2kipYfWjPo0OsneR1l6zDvUCb2aeoswm51mNwHIX2Wt/GRvAp+GSex7qZ5DVIA6wknUugPFH4KqmhMuANs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Scan the PTEs order-wise, using the mask of suitable orders for this VMA derived in conjunction with sysfs THP settings. Scale down the tunables; in case of collapse failure, we drop down to the next order. Otherwise, we try to jump to the highest possible order and then start a fresh scan. Note that madvise(MADV_COLLAPSE) has not been generalized. Signed-off-by: Dev Jain --- mm/khugepaged.c | 84 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 69 insertions(+), 15 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 886c76816963..078794aa3335 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -1111,7 +1112,7 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, } static int collapse_huge_page(struct mm_struct *mm, unsigned long address, - int referenced, int unmapped, + int referenced, int unmapped, int order, struct collapse_control *cc) { LIST_HEAD(compound_pagelist); @@ -1278,38 +1279,59 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, unsigned long address, bool *mmap_locked, struct collapse_control *cc) { - pmd_t *pmd; - pte_t *pte, *_pte; - int result = SCAN_FAIL, referenced = 0; - int none_or_zero = 0, shared = 0; - struct page *page = NULL; + unsigned int max_ptes_shared, max_ptes_none, max_ptes_swap; + int referenced, shared, none_or_zero, unmapped; + unsigned long _address, org_address = address; struct folio *folio = NULL; - unsigned long _address; - spinlock_t *ptl; - int node = NUMA_NO_NODE, unmapped = 0; + struct page *page = NULL; + int node = NUMA_NO_NODE; + int result = SCAN_FAIL; bool writable = false; + unsigned long orders; + pte_t *pte, *_pte; + spinlock_t *ptl; + pmd_t *pmd; + int order; VM_BUG_ON(address & ~HPAGE_PMD_MASK); + orders = thp_vma_allowable_orders(vma, vma->vm_flags, + TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER + 1) - 1); + orders = thp_vma_suitable_orders(vma, address, orders); + order = highest_order(orders); + + /* MADV_COLLAPSE needs to work irrespective of sysfs setting */ + if (!cc->is_khugepaged) + order = HPAGE_PMD_ORDER; + +scan_pte_range: + + max_ptes_shared = khugepaged_max_ptes_shared >> (HPAGE_PMD_ORDER - order); + max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order); + max_ptes_swap = khugepaged_max_ptes_swap >> (HPAGE_PMD_ORDER - order); + referenced = 0, shared = 0, none_or_zero = 0, unmapped = 0; + + /* Check pmd after taking mmap lock */ result = find_pmd_or_thp_or_none(mm, address, &pmd); if (result != SCAN_SUCCEED) goto out; memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); + pte = pte_offset_map_lock(mm, pmd, address, &ptl); if (!pte) { result = SCAN_PMD_NULL; goto out; } - for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; + for (_address = address, _pte = pte; _pte < pte + (1UL << order); _pte++, _address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); if (is_swap_pte(pteval)) { ++unmapped; if (!cc->is_khugepaged || - unmapped <= khugepaged_max_ptes_swap) { + unmapped <= max_ptes_swap) { /* * Always be strict with uffd-wp * enabled swap entries. Please see @@ -1330,7 +1352,7 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, ++none_or_zero; if (!userfaultfd_armed(vma) && (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + none_or_zero <= max_ptes_none)) { continue; } else { result = SCAN_EXCEED_NONE_PTE; @@ -1375,7 +1397,7 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, if (folio_likely_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared) { + shared > max_ptes_shared) { result = SCAN_EXCEED_SHARED_PTE; count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); goto out_unmap; @@ -1432,7 +1454,7 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, result = SCAN_PAGE_RO; } else if (cc->is_khugepaged && (!referenced || - (unmapped && referenced < HPAGE_PMD_NR / 2))) { + (unmapped && referenced < (1UL << order) / 2))) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -1441,9 +1463,41 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { result = collapse_huge_page(mm, address, referenced, - unmapped, cc); + unmapped, order, cc); /* collapse_huge_page will return with the mmap_lock released */ *mmap_locked = false; + + /* Immediately exit on exhaustion of range */ + if (_address == org_address + (PAGE_SIZE << HPAGE_PMD_ORDER)) + goto out; + } + if (result != SCAN_SUCCEED) { + + /* Go to the next order. */ + order = next_order(&orders, order); + if (order < 2) + goto out; + goto maybe_mmap_lock; + } else { + address = _address; + pte = _pte; + + + /* Get highest order possible starting from address */ + order = count_trailing_zeros(address >> PAGE_SHIFT); + + /* This needs to be present in the mask too */ + if (!(orders & (1UL << order))) + order = next_order(&orders, order); + if (order < 2) + goto out; + +maybe_mmap_lock: + if (!(*mmap_locked)) { + mmap_read_lock(mm); + *mmap_locked = true; + } + goto scan_pte_range; } out: trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced,