From patchwork Mon Dec 16 16:50:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B390CE7717F for ; Mon, 16 Dec 2024 16:52:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E7906B00C3; Mon, 16 Dec 2024 11:52:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4961F8D0001; Mon, 16 Dec 2024 11:52:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 337076B00C5; Mon, 16 Dec 2024 11:52:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 12B1C6B00C3 for ; Mon, 16 Dec 2024 11:52:33 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C2F2343D84 for ; Mon, 16 Dec 2024 16:52:32 +0000 (UTC) X-FDA: 82901415114.18.C8CED5B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id 2E2CAA000A for ; Mon, 16 Dec 2024 16:51:50 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367937; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zeIYwTH/nvAgl5ZxraJxiNqlYXUy1Hcqb2mvZuFqH0Y=; b=gSZB8gdMXxSyOB2u/ChD371WcypmV67MOjOtGUKqW8zTtafo19LgvpMSexX36DeAev7gun Vo0rk5gO5AHv/PkhHr0r4wMtnoZdOn3dxskvqUL/S8BFhiIREIR3NJQ0AXKwBhlQpnGDVk uSSLhgXqwCJ3NoIPHXuDxLxUsT3llW0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367937; a=rsa-sha256; cv=none; b=ro6/tWS/XKpGxmWhSqT8P1VEXPFLcggMdkCU958MHde4Hh1ceaPHthu5qSQUPazC5vSBxa uWyH3oNQSjPWsoD9U8IWEzhiDnQo5dGy84DUmdguRPBlAhMFhm93Yq6jc3pJpRplO8VPik D+AVoRumKIzSJKLouMF6EdY5IlpBxFw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 905511AED; Mon, 16 Dec 2024 08:52:58 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0F31C3F528; Mon, 16 Dec 2024 08:52:19 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 05/12] khugepaged: Generalize __collapse_huge_page_isolate() Date: Mon, 16 Dec 2024 22:20:58 +0530 Message-Id: <20241216165105.56185-6-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 2E2CAA000A X-Stat-Signature: fegqadxbuiw58zj19e5zw4p9tqecd1w5 X-Rspam-User: X-HE-Tag: 1734367910-769212 X-HE-Meta: U2FsdGVkX19fO50VTGGc0K2dqIVfLxme+pqC6DB9L39Rj3LPYBKxMvb3xd3eBm05yKCzZKEIO/gDxUgW56y2PCQoKy/Fo/bvXnSMT7ESHBWemrNoykZVEzZ2q5vp/zWr+lqemg1ZPAzM+yj4SFMBetRcG7veHHNkOTwToBX99yB9rq30N4RgGoUmMnPEDNt7f7LsxirVhmkWv/We+BMrowlFpDv9sCebpvkPYNcPRb1hGv3DIIYOkOPQ4GEaljSDv2HYJlZH8U7ATNJ1x9ZAaHRTHCFQX7i0AqDozH5M5xjRML0FSJ98342iBqWaNzGqxEcQE+vZTKN3Q4ab0HO7AQgwYVSez5pF8rMCMsFtLXDHfNfYYRxiG3MmZdjEVe8eVEEVS9RJVYPunwQ/HythgwlvsIQNmLcyZYDDE0OkFVISuk6wBPlYsGsbL3zO4HPLducZ9ER0JZhWWmad1Fy3c/ZNB4BIt4ium02ns7wUqQ2I9sda4PlEY6ydrecs1e7p+nh58owBEyi1kyFuzj1bymFnen7WgbKO9m/BLJjvlY5IgD7BS5UFBx/us/gPbLyt8jGoeb2xTigy+RKOwEOJAhvwIKw8pvyjS8yey7hloDyBPbWzyz/8DiPgVuuNPJpOhOhZXDsOJmEWP0u7jd6pP8VeyUndq5buUP/SvGr5fk+8QC7HeowgFq+S0er5nzad4lG/njt5pcJLv7G0L6nrqTVCr5Q0wiYGwYC5e4/3h2ZfmnYHeSLpWJMrsiNyHajUsPaLTj8WmEWL2Fv9gkvIZIo9AeVYtKKtx/iuJroiIaC244PxcmDcV5jvTuCHVyavfTR+Lynv/3491Ds4ujX5jgF7e3VR+45Bfp0IH0VvfK4BhPrSVD3mIiYMUuYUUkRoJTU+zvCzgOBfTM6zWchsM+0exszAnzRPq1aH9/kRl9gIvZoh8gx7FpBJQ1d4FmKKjJ0I2UkBhzzBVAc6j6d 30Dr2L/1 BOhpvJ2Qsw8v7spx6uDISLf3Lm0JMrOEhz98MFtROsNBPwYmTlC0SA6ay2jOgEJhkpOwwuYonsiBnd+o9ifDhCij2nw+CpYUS5Y4EkJmSr7LOs2PxVe0pExRTojVIn/F41a06AThWtBtnEzu694hHCd1V0ScAdQLN8xwWOzT9SsQRLlsHYMgEmBVPIzC1JFijROPyeWoX6W/sT87doznQ29c+RthL+YIVZ1IQhBKCKXw6YwktHdxJCTzVIeKU2Vc+xWsTaz8yHOBAp/azHDQ8sXmEN59GxI42Nf1oEKtUKrIiKAa6rCA5XR+129hhq9+rPxtl7a5V4huhYT8+jrM2nQZZaTtjx5J0/AC30CY6pBKtwzLND+o2hro2Bx78Pib20Yb3vZJSicpC4T4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Scale down the scan range and the sysfs tunables according to the scan order, and isolate the folios. Signed-off-by: Dev Jain --- mm/khugepaged.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f52dae7d5179..de044b1f83d4 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -564,15 +564,18 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, unsigned long address, pte_t *pte, struct collapse_control *cc, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, int order) { - struct page *page = NULL; - struct folio *folio = NULL; - pte_t *_pte; + unsigned int max_ptes_shared = khugepaged_max_ptes_shared >> (HPAGE_PMD_ORDER - order); + unsigned int max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order); int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; + struct folio *folio = NULL; + struct page *page = NULL; bool writable = false; + pte_t *_pte; - for (_pte = pte; _pte < pte + HPAGE_PMD_NR; + + for (_pte = pte; _pte < pte + (1UL << order); _pte++, address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); if (pte_none(pteval) || (pte_present(pteval) && @@ -580,7 +583,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, ++none_or_zero; if (!userfaultfd_armed(vma) && (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + none_or_zero <= max_ptes_none)) { continue; } else { result = SCAN_EXCEED_NONE_PTE; @@ -609,7 +612,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, if (folio_likely_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared) { + shared > max_ptes_shared) { result = SCAN_EXCEED_SHARED_PTE; count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); goto out; @@ -1200,7 +1203,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); if (pte) { result = __collapse_huge_page_isolate(vma, address, pte, cc, - &compound_pagelist); + &compound_pagelist, order); spin_unlock(pte_ptl); } else { result = SCAN_PMD_NULL;