From patchwork Mon Dec 16 16:50:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910075 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F41D2E7717F for ; Mon, 16 Dec 2024 16:51:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 81A816B00B9; Mon, 16 Dec 2024 11:51:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A30C6B00BA; Mon, 16 Dec 2024 11:51:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5CFC06B00BC; Mon, 16 Dec 2024 11:51:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 30B806B00B9 for ; Mon, 16 Dec 2024 11:51:48 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D3D2281256 for ; Mon, 16 Dec 2024 16:51:47 +0000 (UTC) X-FDA: 82901413518.28.052777D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf18.hostedemail.com (Postfix) with ESMTP id CA4111C000F for ; Mon, 16 Dec 2024 16:51:32 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf18.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367878; a=rsa-sha256; cv=none; b=y2YizJ1G0azN6VwixoF09XS/xwrabQbSsgTa+7JoAe/uIf57Vz0lUSmYUvZaY3vF9F+Yme PLc9OOK41GO+Rf75cbrqELpKUd1FjbGSuk2Kfi4gkF+JkSWuRk7S02GoJXVTFbcLGYUk8K deTA5LDtcbDJmddGEqPP+v2Ak39698Y= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf18.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367878; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=r84CPpOJKA9HTk87cXu4tnsmTbU0FSX6VXRAe09EnOY=; b=L47HzvIpLmpPvuEYW4pwlgnWwHczS97sk7GfuPlltJ2eEntiwniXq8WwCC8paWKhRSgtzm kq7WZF37DykJu0u+zI3IpKS7HbIWrfrfXq74ggD/VAp0nXTsu3BZt8UTQ1nSOZ+XL/fTTm 2mWX6vbW1FhA8YvY/B+CyY6idovDXbU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8152F11FB; Mon, 16 Dec 2024 08:52:13 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AC6DC3F528; Mon, 16 Dec 2024 08:51:34 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 01/12] khugepaged: Rename hpage_collapse_scan_pmd() -> ptes() Date: Mon, 16 Dec 2024 22:20:54 +0530 Message-Id: <20241216165105.56185-2-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Stat-Signature: arqao6d8uko9rj4a74msax8bz9q14owx X-Rspam-User: X-Rspamd-Queue-Id: CA4111C000F X-Rspamd-Server: rspam08 X-HE-Tag: 1734367892-112782 X-HE-Meta: U2FsdGVkX1/TTw0hQ9BBwj7+BA6uwIGmolIdmsFYPEL+0L0CAVFQzvD0Uj7pymCitrwYzvZBVzN7Pi0MX8z9CxuFLLeHzFfOh1WBu4oX/GbOXAIlLoj08tzQ+64cUvy98s1uLwF6hOXid8GsE1uUmXGGWKRXbbNP7yJGWks1nc0VJ5Sefr+WkEzeKFdOlZRjuqvszCH2t9NfLUOu9uXJYzbmPqmTaTBFnxTfJigPrRo3spE7nJyhECsFIScjblm0JizlP/vkhbe77L7RjOSTPKwkMx/nK4JJTf4zik7xrYO2j6no0gm7hiLAckIVpRvIQIsiviJrxR1ZSlgAqWynXEyVnhbIQdr7rjqdPkesYXhTUC1FSkQNNbTEyXkIYRuNcU0vHpsa4cuLKrH4nLgrd/648hIyYb6ZVEQ1vAXexTraOgj02hkYM7mSJkORDRXLUHsPXeiu9APAR6ZzO1h3NKGSnk5Fqe3SmhpMDPWV4KujWO6zcDMWXzuUDv7Msbpz4KN3gWC/qOkIgvpyjyWTbHdG26CPQ48fn3BN0Kk7DqygX0fjLdm5yvbQU1G1RCAskeJC9FTF6gd1T5Oc+HyFtuKNO//N1FfGc3eAF3XutINrDvg6/eNrn7AVXRg5k2i+cQNsx3CGDYzFGzimPjh33gaMpopBspCHfj/FcGEXDmrNNsHUDxu7AZ34olpjYY7F1bKdLVv/nVgTSz8w5Itr9NnADtaiYbASbzyXkzUlYC9ih/cYA4cqeCqvy3pfDearQc5KoSXyY56wqeUX1KOvWPOpBph9pYUOwgANzhEGcc003z8zm4gN3sR2dOfAlaBQ3vlOAt1RSVemgmbADULxxV0WBPS6WrSAimHA9ualHUPo0uKmGVuEGFMGqjkbyOyf5vnoJN0mNAHu4JPEjst5g3jc3mW2de+ckhtnKr7TFqSB0YdLUV5yPGHeYe7Hjwg4zrOHAP+fDv06Jsj73sm /tNTBalB cm33CiUtL8a+u+UkSeMcnkLYJB/g0EygcXs+PIwhSSX/6mDcpRP38kKufBb/34j2UPBxvA6SUNlk9kfAKGf22UXK5RZYwByYHqpoOokbXE0S7VoMOwM0PdCGU/s9b/yy7Ztgo5hk54p+68jQg3oY2chzwjKoAyYZDu2E/gRWIT7sVr8y3bxYTmy7Z0XBoEw0lmlCyu7g71XVESvk6A9LiRZnmT+MqJqmkNbdGhywMIa5NKWeXExr4DKNHfMulGMvvkykpfCW5ENnChQqzoyPDpvNZ2xcMkLKcRAu/MdNgUsyyHz9rHHzc3aW7Gm/S8hfb6s/tCHrABfDaXuXKEmFJZDQpgpvmcGXhK94ZNC8ffUqEVOFASQuS9KVqO1D8gYMb8uXWGuQ+mJkYO60= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Rename prior to generalizing the collapse function. Signed-off-by: Dev Jain --- mm/khugepaged.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 99dc995aac11..95643e6e5f31 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -605,7 +605,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio = page_folio(page); VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); - /* See hpage_collapse_scan_pmd(). */ + /* See hpage_collapse_scan_ptes(). */ if (folio_likely_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && @@ -991,7 +991,7 @@ static int check_pmd_still_valid(struct mm_struct *mm, /* * Bring missing pages in from swap, to complete THP collapse. - * Only done if hpage_collapse_scan_pmd believes it is worthwhile. + * Only done if hpage_collapse_scan_ptes believes it is worthwhile. * * Called and returns without pte mapped or spinlocks held. * Returns result: if not SCAN_SUCCEED, mmap_lock has been released. @@ -1263,7 +1263,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, return result; } -static int hpage_collapse_scan_pmd(struct mm_struct *mm, +static int hpage_collapse_scan_ptes(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long address, bool *mmap_locked, struct collapse_control *cc) @@ -2457,7 +2457,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, mmap_read_unlock(mm); } } else { - *result = hpage_collapse_scan_pmd(mm, vma, + *result = hpage_collapse_scan_ptes(mm, vma, khugepaged_scan.address, &mmap_locked, cc); } @@ -2792,7 +2792,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, cc); fput(file); } else { - result = hpage_collapse_scan_pmd(mm, vma, addr, + result = hpage_collapse_scan_ptes(mm, vma, addr, &mmap_locked, cc); } if (!mmap_locked) From patchwork Mon Dec 16 16:50:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910076 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54BA1E7717F for ; Mon, 16 Dec 2024 16:52:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD9FF6B00BE; Mon, 16 Dec 2024 11:51:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A648D6B00BF; Mon, 16 Dec 2024 11:51:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DDB36B00C0; Mon, 16 Dec 2024 11:51:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6A1A76B00BE for ; Mon, 16 Dec 2024 11:51:59 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 25BBD418AC for ; Mon, 16 Dec 2024 16:51:59 +0000 (UTC) X-FDA: 82901412972.08.F27DD16 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id 59468140011 for ; Mon, 16 Dec 2024 16:51:37 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367903; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QkzU1v8iL0Z+ISbOeDDWbdecNqDcUREegtxV4C6vrZE=; b=bHVy6m6XwWgqsFBA4iy9aeAxWTJ1ySRYcxdGW4jI/vnjGu1iwY9Mh+px7ghP5LW0oTJV/U HIzAV/2E4oXziLLdaEluyJATgAT8ET/wA/VO4S3ACSWF/PI/bsNYqTVqSeZjeTpgSVvdtw JpW5wVGdgTfcHuqkkOfJYsH6mUR0aTA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367903; a=rsa-sha256; cv=none; b=WiORuTvmldoS4z/j0nagKsrRPaIR+vqKRP2pTUDKlIVma3qzuNPqbNuv4tDAJ8XDthQNXn 0QS8gO4MfeKtxs6WqUU5qF21apnvi4Boo1PwsbRtXWAggTCS8pca+o4YxR8H+AbIIJ+W8t /7oby9XISnU1WqGHjj80w0qHSYvd9jM= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C211B1692; Mon, 16 Dec 2024 08:52:24 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0E60D3F528; Mon, 16 Dec 2024 08:51:45 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 02/12] khugepaged: Generalize alloc_charge_folio() Date: Mon, 16 Dec 2024 22:20:55 +0530 Message-Id: <20241216165105.56185-3-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 59468140011 X-Stat-Signature: mwt8antawtwae5bw9eqb5gpctcuq81mt X-Rspam-User: X-HE-Tag: 1734367897-708722 X-HE-Meta: U2FsdGVkX18IgtJBe/bMUK9jLSji0zGZMIslIvi4349ZkpJy8q4fcwF8i3NKK7mB9BF0eXCOIO5ptTkfCZbUBqZ80hL/LLvMKtga9zyo4iYXeL6hjlwz8kzSCbgdtf7ABFeyt+kJOph0+NH41nw1/UqlXnEzUgs8jww8/TVmHRruxveEDOpDkJpmYaxBmuohcivD128sIelzIhsc9uXn2uPQi1/JWBDjvU4EZXuvnNyioQWEIP1fLJP/XGZFXj+TbZhbGNTRebRuTx0dQ2HjpMih1KsgcHLVQemdBeLZX0Tro+i0fhbAm8w0isnzFNCPFAvbtNjeX3Mq6pdCTV7ZE6NSdba3q3J5B117kllDdIGpYIIQ+OdtZ9RBtRt31ER2OquROCMmtACMvl9KiRvlOoj/s+SjzarFCd3r20lEG3Dbj6Gvnu3MFA95+6/PMblRcobxfxHUMSSmBDo5Xhczit0oWJIKQbg6jncLajTMKx9RdldUlQ+X+BkJhkwW58LwhjwIHAlcDLkd4UHX4C0QPudDItg2O1KAe7gciKwNTEKV9FWD3XThHsR95PY45TlDM3DoaNwbXVkKeUIld2rZVuC5PLeQTSbLom6JR/uJ/wQoFPMB8XZVYHXQA7sKWACPqD1DPw4kPOqh1Ok40679HwIiab4KRLCdcy12FHwyAsJ3rVDjjGt74cA9s/0JRlBvY+BjvN4ZbYWYcqJAnCeBYVZdlNhXcSe97xignleqajRpv0TqXZ49OGUxC7H7pVGR6j/t48yVy29GUgBxIh6o57TuhpTlUQPkLRGY76Gag2r0tpgSyrM3NAd03lII9D7BF57FERcTweYHv9XgmVAkxDVACmzRRiyrnfja07hd7yfXmYQALjIq0tg9IQaCmimhuWFw/6xwTgPAFp3RWK2Pybi4zyzuLX8oZxvfnC+PGIR4lvdNknTekOQJ+B69Rx3A6ODn1qzxZKggt54mVpI 1QIcRpNY zv5eK9Rg9i2vXRtXZZMpFMk45ZljxwTuC5VI4RT2vJACAI5zyfsWPbaPSFk6rFlcuXQL0MhrJEMCFMJJntzh2wuc8aiETV/fdyRT9ZvUpeMP66iekOE7c+Ape1YXxmFrIXPeB9dGAqkgF56Af/6Q4lCjjddR/147Wy/ptDniaDxr7IH/IkQ/bgg05gB8ybJIX0rZwzpY2VPUrXKcxvJ3QGdYY13Mb2wsHRfdsBgvNG0RbFcmPYqogOYCOpvcp44HO/DyVmvZk+IfM84lwaLZKJXVMmRZnYrBikU9SFpK4Dgzg7J5IvNqeaydTvlfXC6jKP+4bTgVe2arWAx+YmMNmZsyAd7Ad8I8QvnFotkI6FYStdCVTt+s8FddLQHWEhGLZAUgie1cnT4DrNsE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pass order to alloc_charge_folio() and update mTHP statistics. Signed-off-by: Dev Jain --- include/linux/huge_mm.h | 2 ++ mm/huge_memory.c | 4 ++++ mm/khugepaged.c | 13 +++++++++---- 3 files changed, 15 insertions(+), 4 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 93e509b6c00e..8b6d0fed99b3 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -119,6 +119,8 @@ enum mthp_stat_item { MTHP_STAT_ANON_FAULT_ALLOC, MTHP_STAT_ANON_FAULT_FALLBACK, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, + MTHP_STAT_ANON_COLLAPSE_ALLOC, + MTHP_STAT_ANON_COLLAPSE_ALLOC_FAILED, MTHP_STAT_ZSWPOUT, MTHP_STAT_SWPIN, MTHP_STAT_SWPIN_FALLBACK, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2da5520bfe24..2e582fad4c77 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -615,6 +615,8 @@ static struct kobj_attribute _name##_attr = __ATTR_RO(_name) DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); +DEFINE_MTHP_STAT_ATTR(anon_collapse_alloc, MTHP_STAT_ANON_COLLAPSE_ALLOC); +DEFINE_MTHP_STAT_ATTR(anon_collapse_alloc_failed, MTHP_STAT_ANON_COLLAPSE_ALLOC_FAILED); DEFINE_MTHP_STAT_ATTR(zswpout, MTHP_STAT_ZSWPOUT); DEFINE_MTHP_STAT_ATTR(swpin, MTHP_STAT_SWPIN); DEFINE_MTHP_STAT_ATTR(swpin_fallback, MTHP_STAT_SWPIN_FALLBACK); @@ -636,6 +638,8 @@ static struct attribute *anon_stats_attrs[] = { &anon_fault_alloc_attr.attr, &anon_fault_fallback_attr.attr, &anon_fault_fallback_charge_attr.attr, + &anon_collapse_alloc_attr.attr, + &anon_collapse_alloc_failed_attr.attr, #ifndef CONFIG_SHMEM &zswpout_attr.attr, &swpin_attr.attr, diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 95643e6e5f31..02cd424b8e48 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1073,21 +1073,26 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, } static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, - struct collapse_control *cc) + int order, struct collapse_control *cc) { gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : GFP_TRANSHUGE); int node = hpage_collapse_find_target_node(cc); struct folio *folio; - folio = __folio_alloc(gfp, HPAGE_PMD_ORDER, node, &cc->alloc_nmask); + folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask); if (!folio) { *foliop = NULL; count_vm_event(THP_COLLAPSE_ALLOC_FAILED); + if (order != HPAGE_PMD_ORDER) + count_mthp_stat(order, MTHP_STAT_ANON_COLLAPSE_ALLOC_FAILED); return SCAN_ALLOC_HUGE_PAGE_FAIL; } count_vm_event(THP_COLLAPSE_ALLOC); + if (order != HPAGE_PMD_ORDER) + count_mthp_stat(order, MTHP_STAT_ANON_COLLAPSE_ALLOC); + if (unlikely(mem_cgroup_charge(folio, mm, gfp))) { folio_put(folio); *foliop = NULL; @@ -1124,7 +1129,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, */ mmap_read_unlock(mm); - result = alloc_charge_folio(&folio, mm, cc); + result = alloc_charge_folio(&folio, mm, order, cc); if (result != SCAN_SUCCEED) goto out_nolock; @@ -1850,7 +1855,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem); VM_BUG_ON(start & (HPAGE_PMD_NR - 1)); - result = alloc_charge_folio(&new_folio, mm, cc); + result = alloc_charge_folio(&new_folio, mm, HPAGE_PMD_ORDER, cc); if (result != SCAN_SUCCEED) goto out; From patchwork Mon Dec 16 16:50:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7736CE7717F for ; Mon, 16 Dec 2024 16:52:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 115496B00BF; Mon, 16 Dec 2024 11:52:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 09EAF6B00C0; Mon, 16 Dec 2024 11:52:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5A036B00C1; Mon, 16 Dec 2024 11:52:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C746B6B00BF for ; Mon, 16 Dec 2024 11:52:10 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 900571A0E2F for ; Mon, 16 Dec 2024 16:52:10 +0000 (UTC) X-FDA: 82901413434.01.D150DAC Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id AA788120003 for ; Mon, 16 Dec 2024 16:51:27 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367915; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fK2KeTT9twhZXnPuM8UHgnUOyCXICzvhJCew7lsKW+E=; b=pXzTBjNHo0We8MpkM9JzmyGFN6OA2cuFE9NNfwYCrCFqLsLH4GXlWgigLNCeexdPVfMBfv 9xHxCicRp4KlObjN9rekMUSgdqP8rGaSm9qb1tO2DEX26T/I+TKvKiYulCH/aUvZ6vsEx0 oXxx99YEnJI6fCumePWAPiQODG92e+E= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367915; a=rsa-sha256; cv=none; b=A9ltOOzqMS+JZx2S0SC7X3m+8RSqs2hZkVBCelJi+OjvwFxqDa1NGwUdISgIn1inSPnOu6 EME/hcDBebBr94NqurVw7VJywbeYaTD7Y40C/1RObRjmSrQmRcW87xtyexUSEPzxw0Usan v/L5Fq7YaHUciwTV9grGQgCVt5qGsAU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 4136B16F2; Mon, 16 Dec 2024 08:52:36 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 527BB3F528; Mon, 16 Dec 2024 08:51:57 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 03/12] khugepaged: Generalize hugepage_vma_revalidate() Date: Mon, 16 Dec 2024 22:20:56 +0530 Message-Id: <20241216165105.56185-4-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AA788120003 X-Stat-Signature: x6w6mm7hm8nfsa57yqx9ntzwud4bheaz X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1734367887-724545 X-HE-Meta: U2FsdGVkX196h4dyUQYyCrIeAxbGdB5QRfEZMkLCUaSc38USal6eCb7Xou4Qne4RmfIMif7DRhxpPEIgfEKAzCBoWUnZCBrS10SMVrTke4fkxFcSi9DhdEe6Qj3tsiL3Fv2+QkG+ZZdnolWkGAYxujv1HjqcBcNfxXphYu4I4XdVe2LtL0jAkz09bRLljqUqgC6L5erQKsTjCHuIuiL8MLZykFefoYYJu/LiHJ89DW9n/g4+BCT8uDjv1JCTZotw0Uy4AtIGjDLaLoFMYKfMAzMm1cTyackvnH6+MLnJ+4RtzfeiLB95L9MkPkSTDWjdNmrANqNotG8mcy13Ywg4Bh7coHoHGT+/3F8B1vntd8oFZglPcB+tgPk8S1SmvypJLkgA6XXH1QFKKK3FQ10W6TMl952iTdW9k5oG17Bqf/goHphfiMbekLwQc6j+sVIyzq+hkQTT070xuRE/5B5ZLjHchPGoViJKmS+i+RVEg32nUEzlpUgR7Fh7n8HEmu8FFPvcXl2C7OuzF2/dhjHY8IgsIAjvdRH1/pi9HDToUWHIjUwTRuN+sFdRQcH5ssFkTPQ82MsaRLPYweTygLDqF2RUOhXWoqPvFTYQh1TbCi4BhhWZr+2mRpHx8aGeAxF/S2aFbv9/VWNWSR2sTjZ/iY1kWw22xjCdU/9qmQfWxJUJCcTJ97/fMSNFps4iN2CvNB9aOOWSmHIZs4EoWq1zh8qMn1afojqTUdTqEEhXOvn885pU2+okh5xqIo0aBIcdtumtga0W0CNoZA+t7+7YQO3VZS/3wZhrGsHOVxKGng1B4s10UPrCB2SAJNtCrhJ7Pfn9xoXgEtOo1uWCeCQ3UE4y37qL+vXlWRwlfEIWDQQCcY3uZC5f//2B0SMxQkbSmjPJ9w+3jZ0So9DEutkyLQMsD/thJLOuuMi2SPwyAEariePDNbcbE2DWangntH0fWKFkXVpo1FU8KBdtY8v N7cNgXWX 2Scpy3Z8/fsjy931ccfDF/0mHGeANCEbrZIELWzwtA2IN0MNXBzqfLytPGXEzX2x9uDuhkHsWIG2Gaj/aX/fV5N3jJSYjroNzZHE7JuDw+oDLjgsMo8Wk91rf90yPZrbeciQHE6WtQAOQQQTxsijyQDTAiwH066zOBOgxZ5Ql46LMOz4XmmCSFrp1RDVs3gU3p5Dsg/wxh5xLRSZQBz7oe+GTne4LHjr0msP71JfpCV3PpG4t6ccaSDDvYeI6Q3cg/iHtR3oOZdH9O7HDqSl36M/itSOUb7AYAFEbFQMlDBPUYfRW/+S8SOHVG3P/KAtuiQDM+DfJFEqQmJtsQY8Gj/A3eVW9sA5/zBp3xeZFeJlDVvevgOFR6WA7ycestcNRNy6yhK6OrzzSGz0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Post retaking the lock, it must be checked that the VMA is suitable for our scan order. Hence, generalize hugepage_vma_revalidate(). Signed-off-by: Dev Jain --- mm/khugepaged.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 02cd424b8e48..2f0601795471 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -918,7 +918,7 @@ static int hpage_collapse_find_target_node(struct collapse_control *cc) static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, bool expect_anon, - struct vm_area_struct **vmap, + struct vm_area_struct **vmap, int order, struct collapse_control *cc) { struct vm_area_struct *vma; @@ -931,9 +931,9 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, if (!vma) return SCAN_VMA_NULL; - if (!thp_vma_suitable_order(vma, address, PMD_ORDER)) + if (!thp_vma_suitable_order(vma, address, order)) return SCAN_ADDRESS_RANGE; - if (!thp_vma_allowable_order(vma, vma->vm_flags, tva_flags, PMD_ORDER)) + if (!thp_vma_allowable_order(vma, vma->vm_flags, tva_flags, order)) return SCAN_VMA_CHECK; /* * Anon VMA expected, the address may be unmapped then @@ -1134,7 +1134,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock; mmap_read_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, cc); + result = hugepage_vma_revalidate(mm, address, true, &vma, order, cc); if (result != SCAN_SUCCEED) { mmap_read_unlock(mm); goto out_nolock; @@ -1168,7 +1168,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, * mmap_lock. */ mmap_write_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, cc); + result = hugepage_vma_revalidate(mm, address, true, &vma, order, cc); if (result != SCAN_SUCCEED) goto out_up_write; /* check if the pmd is still valid */ @@ -2776,7 +2776,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, mmap_read_lock(mm); mmap_locked = true; result = hugepage_vma_revalidate(mm, addr, false, &vma, - cc); + HPAGE_PMD_ORDER, cc); if (result != SCAN_SUCCEED) { last_fail = result; goto out_nolock; From patchwork Mon Dec 16 16:50:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C4BBE77180 for ; Mon, 16 Dec 2024 16:52:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A8C356B00C1; Mon, 16 Dec 2024 11:52:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3BA56B00C2; Mon, 16 Dec 2024 11:52:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8DC2B6B00C3; Mon, 16 Dec 2024 11:52:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6DEF86B00C1 for ; Mon, 16 Dec 2024 11:52:22 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1BC791209D4 for ; Mon, 16 Dec 2024 16:52:22 +0000 (UTC) X-FDA: 82901415030.29.431BB69 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 1A4A64001B for ; Mon, 16 Dec 2024 16:51:57 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367921; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FLj2WcjZj7RXOkPAUGtCV/UiktiK1JzYIk2pp7vpSMQ=; b=phXFXvXLysmeyHadW13EuvIq/6ZDu43oCb9Fxy/IVtio6VeCCCJ1dygLXGSKtvCQhA3PAd 6bfhvVds7It6epFTYBuZW9OkaOw5xBnBREWy5kVf4dELZwWckvSSiIPapXoYNTU8wQ0Jfg +Fx1Zjlf77YJuTQQzxKSSmAmpULlwQQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367921; a=rsa-sha256; cv=none; b=eRKsMb8K4qwED22vuKbQtNXKTCANhTBGckviO+HqJNZ4nudQMoC1bQ1nN2Spm3r4jMZTee uXMKuuoLq97jxHHOAyhklo1IjlJztp7srSw0apenVNIRktY7eJx8P1YP2+fmxpUU8It4G8 DSpL0YtxqL2eT/3Is8A+ZBnlnlliIp4= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 81C5F16F3; Mon, 16 Dec 2024 08:52:47 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C5BF03F528; Mon, 16 Dec 2024 08:52:08 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 04/12] khugepaged: Generalize __collapse_huge_page_swapin() Date: Mon, 16 Dec 2024 22:20:57 +0530 Message-Id: <20241216165105.56185-5-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 1A4A64001B X-Stat-Signature: 5ksys34ytzesne8winuc9m577ubzcyqp X-Rspam-User: X-HE-Tag: 1734367917-847994 X-HE-Meta: U2FsdGVkX19+pRpmlc39Adam8bzbcqDA8C/E/BwP6eoeJBiWhhC/DKUSKtd0VEPz3kyG+TMm3dmSPj6RR+g+DY9YWf1vmT/kCJvEZ5YODbW0oKJyaPqKmCeDEq8P2XipRQMn+t1TdHLKhU1yYqn1gBvyNkw/V38N84ibnm++gluTW5nXxqyPFYh8wvU10NuP+aPj/m6BhRvlR9xr6lnWFMzHWeSEGQdaZsQdLcEMKlGtnYOaa8tYfEGO7iI9VvVlregpwU2fjBIQ9t1k3o4CuT1CPGqUgR1R9E44RmVwi81oVZVR59Swenc+TpKdI7QNdAGu41gZlRDHWhP+buvyYY7df8wEPJWzgLgkZo6dX5c+Lwgra8w358a+CcASn8oPTtYXYHtve/cwkVGkrHmups5ItkMGauFsjWmKG1DP4Vr+y8RhT5RAvSFV4bzd0hnrAsvSNK/2QQNmylAeEn1qyzDQ82Bntce9RO+odGrnx8VDC7siO/GU2oGE1bIjccsbF26QxmcTLqDwFzMn1FnQNjX+0D8++JnWWQNHrsGLa+yklSlDb/FOZuelxSDZ1hDdQvHbylktKKWSoGYHb072BJHpj6pGbi5NCBP/mviE8HD7fVELnmZgrnLl3l5ZYwzV7/D1e80X0khutbgaBxwswi968OrzZ86Qv8aP6cLUTJ/TpFCUgeUOofEA5SwUDySOkvL9bH7IRq9adYCqIs5lZ+HO954pioe3c02xygrTuu+dwhSs5/jP3q4b3NGcPGWbyx7MvQQ1KHjFuLrrFtl+XFp9imPuVJlZFSQq6IBsYxqIispdA/LriYEkVJy9gM5u5jy5GszVq2ixoKS4TQ0wtTga+8RIVcn+7rCwM9BA6/gOBYGpWxS6l5cseyGW1wUpMy6rX2CXMfCkmm0fz3ieVR1JxrhEJVqDUYSDWGjKcQleaEZ/hemcopHdP2ox/ITmzGyQhE2LrLTzKqNJdVV jfxeVG/f 7TP+vfOMaXINLqC9lGKnkJTIfPajqGPtJ013QdC1VdWfoC7EMwdDA6eKcHG5YPDZGzdbrBjlK65P8eUkO4hipk/V9H3gHVXssqTUxFqjRDnsG600STyT78aadw9bKx/RMZL7GKagOPkIA1pNhxN3c7dFsmzoTRngTEIpAuhyMh/+msGoNmXJXn9Ghuqcl0JvNqkhIVrbAZv6/aW0qP4z8DViq7EyxGn7+puQRpGIpEaogiTWrnKIuAA8nCs38BRgXedJwdTHb9IF6nQKqNUD3Rvuf5Ypr8eLCaZ9Ur685mikqYKE4eKzgKGCASSKKHjkkRbiA194pRtiF5nKGu5DiVkYyjhnAsgbr79uGoJzvc4MDCEdRjJnsi/83s0UWib7+InHDTQFYWgP3FeQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If any PTE in our scan range is a swap entry, then use do_swap_page() to swap-in the corresponding folio. Signed-off-by: Dev Jain --- mm/khugepaged.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2f0601795471..f52dae7d5179 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -998,17 +998,17 @@ static int check_pmd_still_valid(struct mm_struct *mm, */ static int __collapse_huge_page_swapin(struct mm_struct *mm, struct vm_area_struct *vma, - unsigned long haddr, pmd_t *pmd, - int referenced) + unsigned long addr, pmd_t *pmd, + int referenced, int order) { int swapped_in = 0; vm_fault_t ret = 0; - unsigned long address, end = haddr + (HPAGE_PMD_NR * PAGE_SIZE); + unsigned long address, end = addr + ((1UL << order) * PAGE_SIZE); int result; pte_t *pte = NULL; spinlock_t *ptl; - for (address = haddr; address < end; address += PAGE_SIZE) { + for (address = addr; address < end; address += PAGE_SIZE) { struct vm_fault vmf = { .vma = vma, .address = address, @@ -1153,7 +1153,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, * that case. Continuing to collapse causes inconsistency. */ result = __collapse_huge_page_swapin(mm, vma, address, pmd, - referenced); + referenced, order); if (result != SCAN_SUCCEED) goto out_nolock; } From patchwork Mon Dec 16 16:50:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B390CE7717F for ; Mon, 16 Dec 2024 16:52:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4E7906B00C3; Mon, 16 Dec 2024 11:52:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4961F8D0001; Mon, 16 Dec 2024 11:52:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 337076B00C5; Mon, 16 Dec 2024 11:52:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 12B1C6B00C3 for ; Mon, 16 Dec 2024 11:52:33 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C2F2343D84 for ; Mon, 16 Dec 2024 16:52:32 +0000 (UTC) X-FDA: 82901415114.18.C8CED5B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf15.hostedemail.com (Postfix) with ESMTP id 2E2CAA000A for ; Mon, 16 Dec 2024 16:51:50 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367937; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zeIYwTH/nvAgl5ZxraJxiNqlYXUy1Hcqb2mvZuFqH0Y=; b=gSZB8gdMXxSyOB2u/ChD371WcypmV67MOjOtGUKqW8zTtafo19LgvpMSexX36DeAev7gun Vo0rk5gO5AHv/PkhHr0r4wMtnoZdOn3dxskvqUL/S8BFhiIREIR3NJQ0AXKwBhlQpnGDVk uSSLhgXqwCJ3NoIPHXuDxLxUsT3llW0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367937; a=rsa-sha256; cv=none; b=ro6/tWS/XKpGxmWhSqT8P1VEXPFLcggMdkCU958MHde4Hh1ceaPHthu5qSQUPazC5vSBxa uWyH3oNQSjPWsoD9U8IWEzhiDnQo5dGy84DUmdguRPBlAhMFhm93Yq6jc3pJpRplO8VPik D+AVoRumKIzSJKLouMF6EdY5IlpBxFw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 905511AED; Mon, 16 Dec 2024 08:52:58 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 0F31C3F528; Mon, 16 Dec 2024 08:52:19 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 05/12] khugepaged: Generalize __collapse_huge_page_isolate() Date: Mon, 16 Dec 2024 22:20:58 +0530 Message-Id: <20241216165105.56185-6-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 2E2CAA000A X-Stat-Signature: fegqadxbuiw58zj19e5zw4p9tqecd1w5 X-Rspam-User: X-HE-Tag: 1734367910-769212 X-HE-Meta: U2FsdGVkX19fO50VTGGc0K2dqIVfLxme+pqC6DB9L39Rj3LPYBKxMvb3xd3eBm05yKCzZKEIO/gDxUgW56y2PCQoKy/Fo/bvXnSMT7ESHBWemrNoykZVEzZ2q5vp/zWr+lqemg1ZPAzM+yj4SFMBetRcG7veHHNkOTwToBX99yB9rq30N4RgGoUmMnPEDNt7f7LsxirVhmkWv/We+BMrowlFpDv9sCebpvkPYNcPRb1hGv3DIIYOkOPQ4GEaljSDv2HYJlZH8U7ATNJ1x9ZAaHRTHCFQX7i0AqDozH5M5xjRML0FSJ98342iBqWaNzGqxEcQE+vZTKN3Q4ab0HO7AQgwYVSez5pF8rMCMsFtLXDHfNfYYRxiG3MmZdjEVe8eVEEVS9RJVYPunwQ/HythgwlvsIQNmLcyZYDDE0OkFVISuk6wBPlYsGsbL3zO4HPLducZ9ER0JZhWWmad1Fy3c/ZNB4BIt4ium02ns7wUqQ2I9sda4PlEY6ydrecs1e7p+nh58owBEyi1kyFuzj1bymFnen7WgbKO9m/BLJjvlY5IgD7BS5UFBx/us/gPbLyt8jGoeb2xTigy+RKOwEOJAhvwIKw8pvyjS8yey7hloDyBPbWzyz/8DiPgVuuNPJpOhOhZXDsOJmEWP0u7jd6pP8VeyUndq5buUP/SvGr5fk+8QC7HeowgFq+S0er5nzad4lG/njt5pcJLv7G0L6nrqTVCr5Q0wiYGwYC5e4/3h2ZfmnYHeSLpWJMrsiNyHajUsPaLTj8WmEWL2Fv9gkvIZIo9AeVYtKKtx/iuJroiIaC244PxcmDcV5jvTuCHVyavfTR+Lynv/3491Ds4ujX5jgF7e3VR+45Bfp0IH0VvfK4BhPrSVD3mIiYMUuYUUkRoJTU+zvCzgOBfTM6zWchsM+0exszAnzRPq1aH9/kRl9gIvZoh8gx7FpBJQ1d4FmKKjJ0I2UkBhzzBVAc6j6d 30Dr2L/1 BOhpvJ2Qsw8v7spx6uDISLf3Lm0JMrOEhz98MFtROsNBPwYmTlC0SA6ay2jOgEJhkpOwwuYonsiBnd+o9ifDhCij2nw+CpYUS5Y4EkJmSr7LOs2PxVe0pExRTojVIn/F41a06AThWtBtnEzu694hHCd1V0ScAdQLN8xwWOzT9SsQRLlsHYMgEmBVPIzC1JFijROPyeWoX6W/sT87doznQ29c+RthL+YIVZ1IQhBKCKXw6YwktHdxJCTzVIeKU2Vc+xWsTaz8yHOBAp/azHDQ8sXmEN59GxI42Nf1oEKtUKrIiKAa6rCA5XR+129hhq9+rPxtl7a5V4huhYT8+jrM2nQZZaTtjx5J0/AC30CY6pBKtwzLND+o2hro2Bx78Pib20Yb3vZJSicpC4T4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Scale down the scan range and the sysfs tunables according to the scan order, and isolate the folios. Signed-off-by: Dev Jain --- mm/khugepaged.c | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f52dae7d5179..de044b1f83d4 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -564,15 +564,18 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, unsigned long address, pte_t *pte, struct collapse_control *cc, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, int order) { - struct page *page = NULL; - struct folio *folio = NULL; - pte_t *_pte; + unsigned int max_ptes_shared = khugepaged_max_ptes_shared >> (HPAGE_PMD_ORDER - order); + unsigned int max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order); int none_or_zero = 0, shared = 0, result = SCAN_FAIL, referenced = 0; + struct folio *folio = NULL; + struct page *page = NULL; bool writable = false; + pte_t *_pte; - for (_pte = pte; _pte < pte + HPAGE_PMD_NR; + + for (_pte = pte; _pte < pte + (1UL << order); _pte++, address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); if (pte_none(pteval) || (pte_present(pteval) && @@ -580,7 +583,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, ++none_or_zero; if (!userfaultfd_armed(vma) && (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + none_or_zero <= max_ptes_none)) { continue; } else { result = SCAN_EXCEED_NONE_PTE; @@ -609,7 +612,7 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, if (folio_likely_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared) { + shared > max_ptes_shared) { result = SCAN_EXCEED_SHARED_PTE; count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); goto out; @@ -1200,7 +1203,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); if (pte) { result = __collapse_huge_page_isolate(vma, address, pte, cc, - &compound_pagelist); + &compound_pagelist, order); spin_unlock(pte_ptl); } else { result = SCAN_PMD_NULL; From patchwork Mon Dec 16 16:50:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CBEDE77180 for ; Mon, 16 Dec 2024 16:52:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 311966B00C5; Mon, 16 Dec 2024 11:52:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C2356B00C6; Mon, 16 Dec 2024 11:52:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 162896B00C7; Mon, 16 Dec 2024 11:52:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E961C6B00C5 for ; Mon, 16 Dec 2024 11:52:44 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7071D120D00 for ; Mon, 16 Dec 2024 16:52:44 +0000 (UTC) X-FDA: 82901415366.26.70A02FE Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 71D298001C for ; Mon, 16 Dec 2024 16:51:46 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367948; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=syZ6BkV2LgDcRliK9yoqySwy+/idtulFsqPrRs6j6ho=; b=eRRE5Hpjxp44BfbfSpEEm5Z/TvFMoH2LrHVwXL09zq1tZCYfeKJsxznQY0FENVk8SlzMMT TUJYRd/l8hnjjdMnFQH/HWBENICEFRg8TBcOqtokHKN1+U8uORECoc+r4e4O5N4fCl7wGb 3PixSPXul6MmUde3r0jGG7mf/dEQxlQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367948; a=rsa-sha256; cv=none; b=U7rUfCa8GK+Tx4LOesMmQOgWKKu+AcCAHjdxQ8B/KKI/FpRkqxTZVVNEhynJpitg2ribdK axt4WGz+DVIIdpy7EIGJyxWPEFD2p31cxp2mPfdS++p8yl8tABm8KxcyYfzgu9BgLfEby0 G0NLpcNUdlwSeKGhC1LlDyK8ARu75a8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2894116F8; Mon, 16 Dec 2024 08:53:10 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 364F13F528; Mon, 16 Dec 2024 08:52:30 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 06/12] khugepaged: Generalize __collapse_huge_page_copy_failed() Date: Mon, 16 Dec 2024 22:20:59 +0530 Message-Id: <20241216165105.56185-7-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 71D298001C X-Stat-Signature: 43k3ofcjkcneq7u56jhuygcojhzsbg65 X-Rspam-User: X-HE-Tag: 1734367906-566176 X-HE-Meta: U2FsdGVkX1+1p6CzUMyMiwfMCdiyj6dmyefaxfip/c1SrGzaOOPZctJGm9CnUjTM4VEsZq9sumy0CTBI+2cPnOt2ez6kfGqM+kBlR7ajmdT9UBSzt56rulu0vZ0Wt/YdY32V7Q3KNBJunr/YLoWJ0b3avirT0FdUpzvi4JO8WYGIgQkA8yAfDdunVfBpHezI9CbvINmHQz6dkwtVEl9gki7dqYfcAG9BkGmvdyzEHvMFttp9K9Wor2fCSWXVGarsy4z9f3owx+fvNkcDpx9rUSo49f9IwCF5Bf+hyAhulhpppPMKNhwaHolR1a73HjoFDfhEiwaJ6hUyR2Saj10OrZo4YOHPqjEgwN1YRldwblzQZZbuUFFpFqVB91V7Cu7QUr6rr14xxmKNHDvtzp2w6mj3BrYNrfYk9yvoJdl6qLXTfv2YtrqXWSIG3iUUSUhlfw2+YSYcvesZ6yLGeCLkIE8kJT6T1qgCmGpIoEd4x0Fjyq+7R1FV7tYQO6MeBMAwEUJih39We+REnPtFuTJJENp1uW1eW4cGFZNkM5N05fPsnKSf9k0YohtVvsNQ5yg46A5gogWHGouptNdpvHX6gQ4WcVurZH1waxL1ezr4JmXyUCPbP5/VVujqkUvGoSMjHr7vQ9hAyD7gI8qU3CzH257mEC58l23qSmEIYfnWqPtHHYc7+uy1Q1izS8HJQZR2d/RzLt2oIKMLiwCtHSk88pSAyTsklrz3ktLqgN0PG2g75xB/g0K20IV3rPsHL9A5e8B6syFXBX1IMXSS0kdnDxl4NxaK8ffftbCt8aWs2/cb/v4ygc42aApWAF4wHrI4a0MQGq1UnVYWge934l94lTS35auYUKcpLd+/BtIisplm88l1Z7QDH7iCRoymNWdcrY11pPbYMso/Onipfr4nGO3pH33IJfSS92V3bUyCIr0ZqTSWO9Pp1eD+jzUqdqymdp51gZ6wu/nv1e5PI0l FmFeBT2V yWpw0vsAXqlcSrB9EtQ6DQSjJAGbCuIgeFLE8U3wsmeRw+prapApvYYHjgfqx/Gx8j1rckekkI8KIoHYoGEXnhqeDo2eB69B5lMRsSzVBkspYF6VXMmGdfUWu1NtDidfZE7aEtbXFZrLRNwdYUITMkAFS0VLcB4SYsr/7+6trJYDx+oXYqRzXKJqZtvhPWgGjBgitspP1XqRMQn+JsyQ70CfsGpUjJIfuA47ATFU6ivXjyMzPliwQJUeznRxv64OVwWIIzm8qU69re+xcGGp1ZUQHXvAmHZLtP10QCDVyxWL29XLLN023kIB6Rq8nmUzEuJk2j95YQcc7V5zJ5P7HLeqey524SDsJkti8dWS/JPsgQqzknXQGnYxzmzUVwhu7Dn8os2MfwMh2KtA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Upon failure, we repopulate the PMD in case of PMD-THP collapse. Hence, make this logic specific for PMD case. Signed-off-by: Dev Jain --- mm/khugepaged.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index de044b1f83d4..886c76816963 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -766,7 +766,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, int order) { spinlock_t *pmd_ptl; @@ -776,14 +776,16 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, * pages. Since pages are still isolated and locked here, * acquiring anon_vma_lock_write is unnecessary. */ - pmd_ptl = pmd_lock(vma->vm_mm, pmd); - pmd_populate(vma->vm_mm, pmd, pmd_pgtable(orig_pmd)); - spin_unlock(pmd_ptl); + if (order == HPAGE_PMD_ORDER) { + pmd_ptl = pmd_lock(vma->vm_mm, pmd); + pmd_populate(vma->vm_mm, pmd, pmd_pgtable(orig_pmd)); + spin_unlock(pmd_ptl); + } /* * Release both raw and compound pages isolated * in __collapse_huge_page_isolate. */ - release_pte_pages(pte, pte + HPAGE_PMD_NR, compound_pagelist); + release_pte_pages(pte, pte + (1UL << order), compound_pagelist); } /* @@ -834,7 +836,7 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, compound_pagelist); else __collapse_huge_page_copy_failed(pte, pmd, orig_pmd, vma, - compound_pagelist); + compound_pagelist, order); return result; } From patchwork Mon Dec 16 16:51:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BE79E77180 for ; Mon, 16 Dec 2024 16:52:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 13D0C6B00C7; Mon, 16 Dec 2024 11:52:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C6BC6B00C8; Mon, 16 Dec 2024 11:52:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA9946B00C9; Mon, 16 Dec 2024 11:52:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CACFC6B00C7 for ; Mon, 16 Dec 2024 11:52:55 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 73C38A0CD1 for ; Mon, 16 Dec 2024 16:52:55 +0000 (UTC) X-FDA: 82901416080.18.7E1194B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id D274AC0013 for ; Mon, 16 Dec 2024 16:52:22 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367953; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ThaVIYRuc3vYNUyLNOH6vkpdMl8i5Jto84xNiz/AIKE=; b=Bmvxki87mo39Jg5M6srb7rZxMd9VQY+k/a+IIpatwACBZcSru6HfX824sbxNlNN+DLrcw8 LyLMgg7SjDSIuTj4pBEGEUjr+ddrWBbYe7g7PxcSLfOa1Iernu0FiGvTzS2pvVjkpwEE7U 1jIY+qfA+QoYLRdmZhe0s0iOGB1l1Kg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367953; a=rsa-sha256; cv=none; b=CW9uIqByQ1GAEmq5PYRc2je4weDVyD+ghJcGfaC+CCYIGxvE5HtkHta+p2MmiJjmzha0x6 t0B2cJGevg2cZqRK0HLZGzCAdAq8YHkpxzBsghcGqPPdMkdimGodUwRsSmK1LDTuhknmJx Rq+a++yKtJOXAJQc07JzrcIwnwXa+Ss= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 42BAF16F8; Mon, 16 Dec 2024 08:53:21 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A83103F528; Mon, 16 Dec 2024 08:52:42 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 07/12] khugepaged: Scan PTEs order-wise Date: Mon, 16 Dec 2024 22:21:00 +0530 Message-Id: <20241216165105.56185-8-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: f4frxzgjc8x3rpxdgbnjmo3te5dfi5d9 X-Rspamd-Queue-Id: D274AC0013 X-Rspam-User: X-HE-Tag: 1734367942-999431 X-HE-Meta: U2FsdGVkX19XAo5Of1Fz9Dg4QpHRItZy/A5UqyKTsRtYg0QApzP8gQArr7pUeoVVLJUTZrn4cqx320rskMkK5KOuFj5UBTjxtNl81B3Qt7WPjZ9VVI/rPk12/01oRZf/VNb3IbK3fy1oY1nViDMn5/7fYvzcBWmoixfX4n5gv3pJxC+vnLrVaGbeWUTuikcraaYZjuAnmTv+/JLRQ7/QmoWjjEMgsG1ANrGLhB1Iqx0aR8nAOsEoXMab6bpEwU5l/w7j0gSVNrO5X8sAVgqdJJqVhN5L1qlMVVWrB6VEY+Zv5+RDJgKL5gPxUJq87ERm2nGgqKvI4hEE9iIJRZfqWFlG6RzdvpPkmV1rOmUlaziXbb30aJbZ7VAZXIndLlYJ9KcY4kQzi32oBA1kUhmNuOrYzTG9qcDEgX2t3AunUSQrFzc9RDRxLstW1kOKUmoGgSD6MHdmDX4rZVndW1SdPGv/EwOuc09fs+ccZ9mZTVVVyfFC1EkiqvwtNS9X2//MQPjjBCGg+nprP8gAidmG7L2NrP6fMAucXvojpN4MT+6QK3Vk6HiUkTAv5GbOVs4MHzJyPqU8d+uTxLTZsE+b9L6PWPG0y+5aYv2RC8MfHPxWDkd0784u6M8/ycN9Vn+8ZFdy/b1w1TJOpUvvzv6ZoUbhBcSUdug+PsBzJJAvypzzYcdElUGZOA/fBVQvOyCCjr5FJsgGes3g4uBIlHLy2ph49Gqxv4BO17mWJ7bIC87P27TLsc3sfjtfV2mscgNAKAnh+ZMw6fyc7KZ5xAiHD8d1gEtnCXovWrXFV7C6ZzMSh5l4LPU4gtMF425TXeh8mfVlLZm3og4cycPf7KMv2b3tD5K7uNx+Z0IdwOFjw/r/UXB0oXy7W0TK2NaiuquK/CXcAe3w5ddaitY3QExSzcD7IbRqTGw2ePo3bL6uOAe7ne7h8i8UBCv0xoDCSCNBFii5EngWM2mmtGLPf47 5NsjIndT vzD3bw+3gyJeU29TgSR6q7ivvwGROW9EceytuvMM5HFIzSzwuXCwzIo5ZzXs9LBV9cVifT4yuMpQJ7cUUD5DpiyAooS3hVyXKp67PI2lM1LSNFhQRGpQds2gyW5P5I/tggoWRhPsUGZ+6NeGJE/t9pt6Kz0lCa2MBqaNtux0Y8aO3eWSyqa0eYiO6XedPWHX4sSiqkgfWKh/pDa48xhdE1Fvu2iCg8XLnj3azKuZ7mN1ood3IWn2kipYfWjPo0OsneR1l6zDvUCb2aeoswm51mNwHIX2Wt/GRvAp+GSex7qZ5DVIA6wknUugPFH4KqmhMuANs X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Scan the PTEs order-wise, using the mask of suitable orders for this VMA derived in conjunction with sysfs THP settings. Scale down the tunables; in case of collapse failure, we drop down to the next order. Otherwise, we try to jump to the highest possible order and then start a fresh scan. Note that madvise(MADV_COLLAPSE) has not been generalized. Signed-off-by: Dev Jain --- mm/khugepaged.c | 84 ++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 69 insertions(+), 15 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 886c76816963..078794aa3335 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -1111,7 +1112,7 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, } static int collapse_huge_page(struct mm_struct *mm, unsigned long address, - int referenced, int unmapped, + int referenced, int unmapped, int order, struct collapse_control *cc) { LIST_HEAD(compound_pagelist); @@ -1278,38 +1279,59 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, unsigned long address, bool *mmap_locked, struct collapse_control *cc) { - pmd_t *pmd; - pte_t *pte, *_pte; - int result = SCAN_FAIL, referenced = 0; - int none_or_zero = 0, shared = 0; - struct page *page = NULL; + unsigned int max_ptes_shared, max_ptes_none, max_ptes_swap; + int referenced, shared, none_or_zero, unmapped; + unsigned long _address, org_address = address; struct folio *folio = NULL; - unsigned long _address; - spinlock_t *ptl; - int node = NUMA_NO_NODE, unmapped = 0; + struct page *page = NULL; + int node = NUMA_NO_NODE; + int result = SCAN_FAIL; bool writable = false; + unsigned long orders; + pte_t *pte, *_pte; + spinlock_t *ptl; + pmd_t *pmd; + int order; VM_BUG_ON(address & ~HPAGE_PMD_MASK); + orders = thp_vma_allowable_orders(vma, vma->vm_flags, + TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER + 1) - 1); + orders = thp_vma_suitable_orders(vma, address, orders); + order = highest_order(orders); + + /* MADV_COLLAPSE needs to work irrespective of sysfs setting */ + if (!cc->is_khugepaged) + order = HPAGE_PMD_ORDER; + +scan_pte_range: + + max_ptes_shared = khugepaged_max_ptes_shared >> (HPAGE_PMD_ORDER - order); + max_ptes_none = khugepaged_max_ptes_none >> (HPAGE_PMD_ORDER - order); + max_ptes_swap = khugepaged_max_ptes_swap >> (HPAGE_PMD_ORDER - order); + referenced = 0, shared = 0, none_or_zero = 0, unmapped = 0; + + /* Check pmd after taking mmap lock */ result = find_pmd_or_thp_or_none(mm, address, &pmd); if (result != SCAN_SUCCEED) goto out; memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); + pte = pte_offset_map_lock(mm, pmd, address, &ptl); if (!pte) { result = SCAN_PMD_NULL; goto out; } - for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; + for (_address = address, _pte = pte; _pte < pte + (1UL << order); _pte++, _address += PAGE_SIZE) { pte_t pteval = ptep_get(_pte); if (is_swap_pte(pteval)) { ++unmapped; if (!cc->is_khugepaged || - unmapped <= khugepaged_max_ptes_swap) { + unmapped <= max_ptes_swap) { /* * Always be strict with uffd-wp * enabled swap entries. Please see @@ -1330,7 +1352,7 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, ++none_or_zero; if (!userfaultfd_armed(vma) && (!cc->is_khugepaged || - none_or_zero <= khugepaged_max_ptes_none)) { + none_or_zero <= max_ptes_none)) { continue; } else { result = SCAN_EXCEED_NONE_PTE; @@ -1375,7 +1397,7 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, if (folio_likely_mapped_shared(folio)) { ++shared; if (cc->is_khugepaged && - shared > khugepaged_max_ptes_shared) { + shared > max_ptes_shared) { result = SCAN_EXCEED_SHARED_PTE; count_vm_event(THP_SCAN_EXCEED_SHARED_PTE); goto out_unmap; @@ -1432,7 +1454,7 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, result = SCAN_PAGE_RO; } else if (cc->is_khugepaged && (!referenced || - (unmapped && referenced < HPAGE_PMD_NR / 2))) { + (unmapped && referenced < (1UL << order) / 2))) { result = SCAN_LACK_REFERENCED_PAGE; } else { result = SCAN_SUCCEED; @@ -1441,9 +1463,41 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, pte_unmap_unlock(pte, ptl); if (result == SCAN_SUCCEED) { result = collapse_huge_page(mm, address, referenced, - unmapped, cc); + unmapped, order, cc); /* collapse_huge_page will return with the mmap_lock released */ *mmap_locked = false; + + /* Immediately exit on exhaustion of range */ + if (_address == org_address + (PAGE_SIZE << HPAGE_PMD_ORDER)) + goto out; + } + if (result != SCAN_SUCCEED) { + + /* Go to the next order. */ + order = next_order(&orders, order); + if (order < 2) + goto out; + goto maybe_mmap_lock; + } else { + address = _address; + pte = _pte; + + + /* Get highest order possible starting from address */ + order = count_trailing_zeros(address >> PAGE_SHIFT); + + /* This needs to be present in the mask too */ + if (!(orders & (1UL << order))) + order = next_order(&orders, order); + if (order < 2) + goto out; + +maybe_mmap_lock: + if (!(*mmap_locked)) { + mmap_read_lock(mm); + *mmap_locked = true; + } + goto scan_pte_range; } out: trace_mm_khugepaged_scan_pmd(mm, &folio->page, writable, referenced, From patchwork Mon Dec 16 16:51:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCD85E7717F for ; Mon, 16 Dec 2024 16:53:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6BC1F6B0082; Mon, 16 Dec 2024 11:53:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 66D416B0083; Mon, 16 Dec 2024 11:53:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5340F6B00C8; Mon, 16 Dec 2024 11:53:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3368A6B0082 for ; Mon, 16 Dec 2024 11:53:07 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C9ACE4575D for ; Mon, 16 Dec 2024 16:53:06 +0000 (UTC) X-FDA: 82901416332.26.FF3CC72 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 4692F40011 for ; Mon, 16 Dec 2024 16:52:52 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367971; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n8GzYR6cW/SAemjTI3W4Wft8Ia3ux1GWyb+HizoSsdY=; b=ofn2fxpW8hhg51YS/pv2rjZcTrKxMIV1Y7jfdCOTfoBKxAxpawfI+VejF0OXlkGCSXJqtG LhXxEpVZiwOEfuLaYFSB1v0dv05tD4deM7QnnxZ2WmDfx0dcRckR57SGWX3ATPB9V1bZK+ uL+qOuKM2jyOJ5IHngXv3lU0PSPyYXU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367971; a=rsa-sha256; cv=none; b=HBl8GwEsW4dKYyudk7n7fgu/8yWMUB1Yvwz5il/1KuV5UhrYorJtCxWsFSYp5fZuQhgagt aMOyC+83+20rcQ6fQhAzXS3BaIr+CRM/022guibXP591sTTF28uKVKvvZLmsqCFtIzN1f5 Yx2ndAtxdBE5QBqOsFiYJuF5q/HvLYs= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8BFE216F8; Mon, 16 Dec 2024 08:53:32 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C07E13F528; Mon, 16 Dec 2024 08:52:53 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 08/12] khugepaged: Abstract PMD-THP collapse Date: Mon, 16 Dec 2024 22:21:01 +0530 Message-Id: <20241216165105.56185-9-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Stat-Signature: 5aq8hq9n7c1frwyb6qguhm6uj7m6dkkw X-Rspamd-Queue-Id: 4692F40011 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1734367972-246123 X-HE-Meta: U2FsdGVkX19OWCZs+AZKWe5RarJkvMukz0mZXQ91UOEphXBE6g6arAP8JNcfWhO0hUOMHkSkJ878pqGmuiyt2Xxw/W7ZnbjrW3i5//GO2hSeQ7Zq6Qg9Wl3m1UASePmDBoxzfnIAuy7np8QSF3hVMukel/JmerwjjBsz+Tv1xSQMvbjcmoseq69PtDE5eVYnuafZKCNblCTANNmQ1yWra+IK5z8rEKGmv6sbxOjmgKUi7wcwx6nsYXrnzpF21EjqXlfCPzpgScFPVFZeLy1RvCPXfIwSVK+1Ihn+ebnHFBivayvyPwUSY6GLmGm//b90/sc5mjEwLo4ABNc2xsVFrV8P0XBbU75Ju4rEgBPeTJDbGOYDS8oApCD1YzqrITWS6u8FnMNaE46WdJd42kNV1BzOnXgPOLpObxIXrlAfIMbD32aYoT+17FjCFIvTbUS4e4N1Xu6JuRqB4g0huboy5WNZ2aMUx9OEIl0A920dKWwmff9ik3RqyotaAFY320BLg+pNQb7gIM+DLTW/BYqaZvLzf9CjSQAVyonbAdW+jd5lrkpZLY7wArqzUNVVlR4frxNm31SeEUanIQkBT8S65xAqrmbFRRqIqpkndCdKIt7QXpjiPCX0kVBj6VMKUq2XfBxu6g/H69UqGh+lF+bZIZgCkgfTW7SL4bzFn5aIjcVL6MuPeO7FzLUT2QPkIpmvgwQRbbaZLQFRjCTHgeU1zqc9x0GD36jzEJch5UjB/XU+Cal0+X9CeNosn8aeQaTxom/y9N/UslTBHOdwncTsr2etXP2GJwkRUkDZN8RDvnIlV5cvjDIHRxLOdehCDNghpzB3miV3Nlepirdr4y2PNBOEgM35MFqnsxePRbPFz85407xwp8JIjUZd2f4IhfMDI1kC8MVa9ZaJM6QTEkdjsRA/423kvYJNUd3E1zx7P1z9RC6TX+Lq6DCi+Uv6cGsGLQKk9EnTbqPaDTd+csH yfMdtEJa b3R3pUpSkWCgK/kVtZFeHKq2pnWnIdYJj25fbSq9jsphkxDTctSWypYD5J/aHxi6o677+3yeaEEpoFQPe6T/zELurfa97RsmyxZOknROynfmcKT/JE71T9eiAax3Vfi/RtECQxSGwRxU6OhrfOMP46qzl/pxCf5GXeSeuje6xVD9RD00HN9KTZ9nTi8FBfiUwZ49zefX9UW/OOHuxib6yV+s0moWjYkAPKPbJN0jp9XUFJ6wcqg1HGYRh6jKrGdJfHX84HToo27vjRXxRJQ6SCPs+i4lLCNYMI2sqKFgpXKsvjvk86G3whLhX9hOP/2ETiY8j X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Abstract away taking the mmap_lock exclusively, copying page contents, and setting the PMD, into vma_collapse_anon_folio_pmd(). Signed-off-by: Dev Jain --- mm/khugepaged.c | 119 +++++++++++++++++++++++++++--------------------- 1 file changed, 66 insertions(+), 53 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 078794aa3335..88beebef773e 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1111,58 +1111,17 @@ static int alloc_charge_folio(struct folio **foliop, struct mm_struct *mm, return SCAN_SUCCEED; } -static int collapse_huge_page(struct mm_struct *mm, unsigned long address, - int referenced, int unmapped, int order, - struct collapse_control *cc) +static int vma_collapse_anon_folio_pmd(struct mm_struct *mm, unsigned long address, + struct vm_area_struct *vma, struct collapse_control *cc, pmd_t *pmd, + struct folio *folio) { + struct mmu_notifier_range range; + spinlock_t *pmd_ptl, *pte_ptl; LIST_HEAD(compound_pagelist); - pmd_t *pmd, _pmd; - pte_t *pte; pgtable_t pgtable; - struct folio *folio; - spinlock_t *pmd_ptl, *pte_ptl; - int result = SCAN_FAIL; - struct vm_area_struct *vma; - struct mmu_notifier_range range; - - VM_BUG_ON(address & ~HPAGE_PMD_MASK); - - /* - * Before allocating the hugepage, release the mmap_lock read lock. - * The allocation can take potentially a long time if it involves - * sync compaction, and we do not need to hold the mmap_lock during - * that. We will recheck the vma after taking it again in write mode. - */ - mmap_read_unlock(mm); - - result = alloc_charge_folio(&folio, mm, order, cc); - if (result != SCAN_SUCCEED) - goto out_nolock; - - mmap_read_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, order, cc); - if (result != SCAN_SUCCEED) { - mmap_read_unlock(mm); - goto out_nolock; - } - - result = find_pmd_or_thp_or_none(mm, address, &pmd); - if (result != SCAN_SUCCEED) { - mmap_read_unlock(mm); - goto out_nolock; - } - - if (unmapped) { - /* - * __collapse_huge_page_swapin will return with mmap_lock - * released when it fails. So we jump out_nolock directly in - * that case. Continuing to collapse causes inconsistency. - */ - result = __collapse_huge_page_swapin(mm, vma, address, pmd, - referenced, order); - if (result != SCAN_SUCCEED) - goto out_nolock; - } + int result; + pmd_t _pmd; + pte_t *pte; mmap_read_unlock(mm); /* @@ -1174,7 +1133,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, * mmap_lock. */ mmap_write_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, order, cc); + + result = hugepage_vma_revalidate(mm, address, true, &vma, HPAGE_PMD_ORDER, cc); if (result != SCAN_SUCCEED) goto out_up_write; /* check if the pmd is still valid */ @@ -1206,7 +1166,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); if (pte) { result = __collapse_huge_page_isolate(vma, address, pte, cc, - &compound_pagelist, order); + &compound_pagelist, HPAGE_PMD_ORDER); spin_unlock(pte_ptl); } else { result = SCAN_PMD_NULL; @@ -1262,11 +1222,64 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, deferred_split_folio(folio, false); spin_unlock(pmd_ptl); - folio = NULL; - result = SCAN_SUCCEED; out_up_write: mmap_write_unlock(mm); + return result; +} + +static int collapse_huge_page(struct mm_struct *mm, unsigned long address, + int referenced, int unmapped, int order, + struct collapse_control *cc) +{ + struct vm_area_struct *vma; + int result = SCAN_FAIL; + struct folio *folio; + pmd_t *pmd; + + /* + * Before allocating the hugepage, release the mmap_lock read lock. + * The allocation can take potentially a long time if it involves + * sync compaction, and we do not need to hold the mmap_lock during + * that. We will recheck the vma after taking it again in write mode. + */ + mmap_read_unlock(mm); + + result = alloc_charge_folio(&folio, mm, order, cc); + if (result != SCAN_SUCCEED) + goto out_nolock; + + mmap_read_lock(mm); + result = hugepage_vma_revalidate(mm, address, true, &vma, order, cc); + if (result != SCAN_SUCCEED) { + mmap_read_unlock(mm); + goto out_nolock; + } + + result = find_pmd_or_thp_or_none(mm, address, &pmd); + if (result != SCAN_SUCCEED) { + mmap_read_unlock(mm); + goto out_nolock; + } + + if (unmapped) { + /* + * __collapse_huge_page_swapin will return with mmap_lock + * released when it fails. So we jump out_nolock directly in + * that case. Continuing to collapse causes inconsistency. + */ + result = __collapse_huge_page_swapin(mm, vma, address, pmd, + referenced, order); + if (result != SCAN_SUCCEED) + goto out_nolock; + } + + if (order == HPAGE_PMD_ORDER) + result = vma_collapse_anon_folio_pmd(mm, address, vma, cc, pmd, folio); + + if (result == SCAN_SUCCEED) + folio = NULL; + out_nolock: if (folio) folio_put(folio); From patchwork Mon Dec 16 16:51:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B6A8E77180 for ; Mon, 16 Dec 2024 16:53:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3EE58D0005; Mon, 16 Dec 2024 11:53:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C7478D0001; Mon, 16 Dec 2024 11:53:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F7898D0005; Mon, 16 Dec 2024 11:53:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5C6348D0001 for ; Mon, 16 Dec 2024 11:53:19 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 1302D1C83CD for ; Mon, 16 Dec 2024 16:53:19 +0000 (UTC) X-FDA: 82901416542.06.913AE45 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id A0766180007 for ; Mon, 16 Dec 2024 16:52:55 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rM3pVA3HxY3BNxKbRTnafuvJLqn04TkiBEKmftfZJkc=; b=omA7CYfmfmorsSI/T9f00ctussouE02qKNvuMX38ZXvFSJ4gnSPZiAV2AVW7hNQMMHSldD vj2ZhkmWpoLd+DnzMaGhzmFMFicHEEBFGgIRiAnajy/s/o1RzwyeB922q2Fq1CTcLIMFHK y0wYun8AKdyxoZSGLrL9O7056Y/p8/Y= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367974; a=rsa-sha256; cv=none; b=hLP7hrUFBlhlXAkvyU8S8auZF1LvFE/INoQ5TZHnFd/kuml2cSjuUvoHkzYlrvjDHJ02GM nxAFXMkxNhnXmK9kC1jd1qSPzTF6NnuKClQ2+BUsTsHqdJoMzemhJkwFiMhkwFssC4tk9P sEUCrXL0Ljg508nT4OSaMryywWmLeEw= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CAC2C16F8; Mon, 16 Dec 2024 08:53:43 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1C5503F528; Mon, 16 Dec 2024 08:53:04 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 09/12] khugepaged: Introduce vma_collapse_anon_folio() Date: Mon, 16 Dec 2024 22:21:02 +0530 Message-Id: <20241216165105.56185-10-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A0766180007 X-Rspamd-Server: rspam12 X-Stat-Signature: jj1rq6i8wtpgadfzb3o8uwe93djk13ay X-Rspam-User: X-HE-Tag: 1734367975-480054 X-HE-Meta: U2FsdGVkX19ouPhnk0B+SegFYhRn9xRqnO0wuDmlX218TDoMiPFlAjNDby60nMlx0Rrl9deAQqTAtDq+flQcjP1342AKXHFS/slXnCk8f9AYCm24v5EGbF7+Jui0a4V8wJdaY16ljEBdXFKw096OnKnlFFur+S+NmhyTVARpNTC3gbI5c3sAAothudOBykSaEb43ITRo3+T0+/QSbAFmFTiznQ9rowCa3H57Ysk0PWho4f41GODqz/83t5nKbtdhO5wrS6EVAgyb90JumsvdYpuImS8+3CZaKnm6BBnOwry4yO0nvaZpiqeCB/dpWby1a3lny9l8U/DMtVQpCJqfnrztSZjWTLz/0PwYZCx0zZgiqn0xX/Jp+582r6r+xqevZT1NYIU37cmjFA5TYy3gPE4DHD1ANVB3Rxh9Td8aZHiu/E5dqRtd5FUdZUaNKiTPejJCec6bNQ7xmD6kRzMt7/taOh2x/Fi2x6+ldM9OvbnOora2KI8YUbpCep1vahlOv9nSdp8bq1aPtFSMUZzYCJcZT3BBFj0B+ErqvygLeMy7Lyw7gshTdyGYOOXxJ8lpQnPwvr/5QVUwqJyiDK/Ymy9weL7GbbT/qVeROig+wRh7oZFJC5kXYnbUbuI5Ght7SZQUJJOpnBp2QDCt/T/hDA3TyE9lwlymUTA9gahd8vRbEOaVe/UkcIS1eAooaxvX7yzqxzCoHpADzLtqdAQES4fVf/LtI9Bsd2r1sUTpoooTQsrpDDxETjjXq85glGNrR0nNGW7Xstx/Scwqn2Pj7jZDrLCSQGxkMfdAp1BSqvfmlWmF9GCd1aLmaGxoNWnibbSSJrj3r3hETYff7TWtqF0RuIzi5XJQ1G4GSrEnLfhN8S6ZXeO2nOlreWH3PChbJ37amDQNtvAoYI4hgd7rIu5HeqneG3zPPgu6eS56B3rrNx9IdseTP3qXvjZpbDhFwobJu68e7jCHacclmpR sMmvZRJL z6S05s/PoRfCjWR/J5+CVLR0eOsIqfO7/Y6o8L5VW9mqV8wVO+jguxPEbGRcftWM7fnrIXZwiJx+BU/6d5BeNPeAr3tTPGgmn3t0cEeG7ImhbH5fXLu9RTjDl4YObVKFBAHJPMSEmrQqLicjErIV8bBHNUJ2KHiZ2oYYv/4ZwriHYbMY35C0rNbZwEXPFX6KZUsVsHxdVYJaaZbql5tPfe/IGij1H9EmnMUBPUGNz8qWLGU8J34152iugMLsGbUFdb8wa2pyTYmHnwcE5HVOjC/Z0KI47Y1f+VCvgx55Ccqso5759Rbm7VXn49Hl6cxS02k+bkEtbNQfZJ102YiLV1FPjihMr1hCAkl+xryZzcbZvtJ4Glho0kR+WWF5nTuIqAx77mDR9VfWCVpbftuBRnuEo8HyHLHUz1Fjm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In contrast to PMD-collapse, we do not need to operate on two levels of pagetable simultaneously. Therefore, downgrade the mmap lock from write to read mode. Still take the anon_vma lock in exclusive mode so as to not waste time in the rmap path, which is anyways going to fail since the PTEs are going to be changed. Under the PTL, copy page contents, clear the PTEs, remove folio pins, and (try to) unmap the old folios. Set the PTEs to the new folio using the set_ptes() API. Signed-off-by: Dev Jain --- Note: I have been trying hard to get rid of the locks in here: we still are taking the PTL around the page copying; dropping the PTL and taking it after the copying should lead to a deadlock, for example: khugepaged madvise(MADV_COLD) folio_lock() lock(ptl) lock(ptl) folio_lock() We can create a locked folio list, altogether drop both the locks, take the PTL, do everything which __collapse_huge_page_isolate() does *except* the isolation and again try locking folios, but then it will reduce efficiency of khugepaged and almost looks like a forced solution :) Please note the following discussion if anyone is interested: https://lore.kernel.org/all/66bb7496-a445-4ad7-8e56-4f2863465c54@arm.com/ (Apologies for not CCing the mailing list from the start) mm/khugepaged.c | 108 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 87 insertions(+), 21 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 88beebef773e..8040b130e677 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -714,24 +714,28 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, struct vm_area_struct *vma, unsigned long address, spinlock_t *ptl, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, int order) { struct folio *src, *tmp; pte_t *_pte; pte_t pteval; - for (_pte = pte; _pte < pte + HPAGE_PMD_NR; + for (_pte = pte; _pte < pte + (1UL << order); _pte++, address += PAGE_SIZE) { pteval = ptep_get(_pte); if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) { add_mm_counter(vma->vm_mm, MM_ANONPAGES, 1); if (is_zero_pfn(pte_pfn(pteval))) { - /* - * ptl mostly unnecessary. - */ - spin_lock(ptl); - ptep_clear(vma->vm_mm, address, _pte); - spin_unlock(ptl); + if (order == HPAGE_PMD_ORDER) { + /* + * ptl mostly unnecessary. + */ + spin_lock(ptl); + ptep_clear(vma->vm_mm, address, _pte); + spin_unlock(ptl); + } else { + ptep_clear(vma->vm_mm, address, _pte); + } ksm_might_unmap_zero_page(vma->vm_mm, pteval); } } else { @@ -740,15 +744,20 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, src = page_folio(src_page); if (!folio_test_large(src)) release_pte_folio(src); - /* - * ptl mostly unnecessary, but preempt has to - * be disabled to update the per-cpu stats - * inside folio_remove_rmap_pte(). - */ - spin_lock(ptl); - ptep_clear(vma->vm_mm, address, _pte); - folio_remove_rmap_pte(src, src_page, vma); - spin_unlock(ptl); + if (order == HPAGE_PMD_ORDER) { + /* + * ptl mostly unnecessary, but preempt has to + * be disabled to update the per-cpu stats + * inside folio_remove_rmap_pte(). + */ + spin_lock(ptl); + ptep_clear(vma->vm_mm, address, _pte); + folio_remove_rmap_pte(src, src_page, vma); + spin_unlock(ptl); + } else { + ptep_clear(vma->vm_mm, address, _pte); + folio_remove_rmap_pte(src, src_page, vma); + } free_page_and_swap_cache(src_page); } } @@ -807,7 +816,7 @@ static void __collapse_huge_page_copy_failed(pte_t *pte, static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, pmd_t *pmd, pmd_t orig_pmd, struct vm_area_struct *vma, unsigned long address, spinlock_t *ptl, - struct list_head *compound_pagelist) + struct list_head *compound_pagelist, int order) { unsigned int i; int result = SCAN_SUCCEED; @@ -815,7 +824,7 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, /* * Copying pages' contents is subject to memory poison at any iteration. */ - for (i = 0; i < HPAGE_PMD_NR; i++) { + for (i = 0; i < (1 << order); i++) { pte_t pteval = ptep_get(pte + i); struct page *page = folio_page(folio, i); unsigned long src_addr = address + i * PAGE_SIZE; @@ -834,7 +843,7 @@ static int __collapse_huge_page_copy(pte_t *pte, struct folio *folio, if (likely(result == SCAN_SUCCEED)) __collapse_huge_page_copy_succeeded(pte, vma, address, ptl, - compound_pagelist); + compound_pagelist, order); else __collapse_huge_page_copy_failed(pte, pmd, orig_pmd, vma, compound_pagelist, order); @@ -1196,7 +1205,7 @@ static int vma_collapse_anon_folio_pmd(struct mm_struct *mm, unsigned long addre result = __collapse_huge_page_copy(pte, folio, pmd, _pmd, vma, address, pte_ptl, - &compound_pagelist); + &compound_pagelist, HPAGE_PMD_ORDER); pte_unmap(pte); if (unlikely(result != SCAN_SUCCEED)) goto out_up_write; @@ -1228,6 +1237,61 @@ static int vma_collapse_anon_folio_pmd(struct mm_struct *mm, unsigned long addre return result; } +/* Enter with mmap read lock */ +static int vma_collapse_anon_folio(struct mm_struct *mm, unsigned long address, + struct vm_area_struct *vma, struct collapse_control *cc, pmd_t *pmd, + struct folio *folio, int order) +{ + int result; + struct mmu_notifier_range range; + spinlock_t *pte_ptl; + LIST_HEAD(compound_pagelist); + pte_t *pte; + pte_t entry; + int nr_pages = folio_nr_pages(folio); + + anon_vma_lock_write(vma->anon_vma); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, mm, address, + address + (PAGE_SIZE << order)); + mmu_notifier_invalidate_range_start(&range); + + pte = pte_offset_map_lock(mm, pmd, address, &pte_ptl); + if (pte) + result = __collapse_huge_page_isolate(vma, address, pte, cc, + &compound_pagelist, order); + else + result = SCAN_PMD_NULL; + + if (unlikely(result != SCAN_SUCCEED)) + goto out_up_read; + + anon_vma_unlock_write(vma->anon_vma); + + __folio_mark_uptodate(folio); + entry = mk_pte(&folio->page, vma->vm_page_prot); + entry = maybe_mkwrite(entry, vma); + + result = __collapse_huge_page_copy(pte, folio, pmd, *pmd, + vma, address, pte_ptl, + &compound_pagelist, order); + if (unlikely(result != SCAN_SUCCEED)) + goto out_up_read; + + folio_ref_add(folio, nr_pages - 1); + folio_add_new_anon_rmap(folio, vma, address, RMAP_EXCLUSIVE); + folio_add_lru_vma(folio, vma); + deferred_split_folio(folio, false); + set_ptes(mm, address, pte, entry, nr_pages); + update_mmu_cache_range(NULL, vma, address, pte, nr_pages); + pte_unmap_unlock(pte, pte_ptl); + mmu_notifier_invalidate_range_end(&range); + result = SCAN_SUCCEED; + +out_up_read: + mmap_read_unlock(mm); + return result; +} + static int collapse_huge_page(struct mm_struct *mm, unsigned long address, int referenced, int unmapped, int order, struct collapse_control *cc) @@ -1276,6 +1340,8 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, if (order == HPAGE_PMD_ORDER) result = vma_collapse_anon_folio_pmd(mm, address, vma, cc, pmd, folio); + else + result = vma_collapse_anon_folio(mm, address, vma, cc, pmd, folio, order); if (result == SCAN_SUCCEED) folio = NULL; From patchwork Mon Dec 16 16:51:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4B1DE7717F for ; Mon, 16 Dec 2024 16:53:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5396D6B0093; Mon, 16 Dec 2024 11:53:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E6BF6B009C; Mon, 16 Dec 2024 11:53:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 338DB6B0096; Mon, 16 Dec 2024 11:53:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 13A118D0001 for ; Mon, 16 Dec 2024 11:53:30 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 88ADB1405C4 for ; Mon, 16 Dec 2024 16:53:29 +0000 (UTC) X-FDA: 82901417886.29.A24B850 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf17.hostedemail.com (Postfix) with ESMTP id 5AED740003 for ; Mon, 16 Dec 2024 16:53:05 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf17.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367994; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=39qRkFxeQ7cExo69lLQfkW13STIJ4xIJ5DI6eTVjZ7A=; b=W1YM5oqpiCYN3qBHEaDn95PKSCxrrTnlUEWcOXa4FmT+/ChwiayUm9b0ujCYeeWCyZHCxM MjI4kjJVZZuQumcpNcSNgtE8Q8RfT7KMdAZcfo+3SqJ86Qx/qTaf01iIx3ItJwFapIseR1 ta1JWbenOvVyRpJHJ4EdN3+URBGJN8I= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf17.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367994; a=rsa-sha256; cv=none; b=7PiS/1r+7wHavmrY/4OmRVW3CHkigu+pBGKKgkfqBqyoShfvNSQDvT1jlpuL/GI00zrGia 9E24HY2uD52FCgg9U184vQ7JpFN0rHRgEt4kX0KIHcog0qFTtdUIifSfS4+pEmTbAy2AH2 CbZo4zwXXHKBEUCEJ7dm/TKyFXhUqc0= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 32FFE16F8; Mon, 16 Dec 2024 08:53:55 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 552383F528; Mon, 16 Dec 2024 08:53:16 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 10/12] khugepaged: Skip PTE range if a larger mTHP is already mapped Date: Mon, 16 Dec 2024 22:21:03 +0530 Message-Id: <20241216165105.56185-11-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5AED740003 X-Stat-Signature: 41k5utfpmte69w19cmw4k1korqdjcezg X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1734367985-221203 X-HE-Meta: U2FsdGVkX1/VeAx9Jnpzwmp7IZEtXqWoQjsexgQOAgaJYtQngzw7f+nMXGZEb8hmcFtSW939yop9CanUWBmtLteTolVTLTBcVN7ow41pyAJdI6EfUcd1SAcMwLUha0V2qW7xi0dYQiuXGc744TBS2XjTYWciHcU4FxMFXkD5J7DkeWhgAIqxnN93AC3Pj+ObC2CRnwh5yx+WPI3jC5zLEnBoiM+J0qaqFX0aTpwQ2n/fcjXlG6H/SrCdItgLwezcrn9arLWXFTQWrEdcScvWCwAm5tKmIMIikuCqicjj4I9vP2gUfkdHgIbbf3K8fSusFbVyVrMAKrZ9jKKYfoSREyTT1GAd+c1rqwrtqTtCqLPw5jtPkHeWWRMADhPNZT9dnebEYuA082BmITHuFriahkuWAo/O3+sxYr2u/FpKOIeL3hAGPUcr1oYzJA3e0whrYvNQtLuVI3cV1zqvZq65OTey25XzJmC8YYBxMr9lUhQMIMxtMlAkm9jALRQnhWr09vYwhW0os15WIQBgIYkJkZJAy7qXVDAl8G1CWORVDB318PhhWGSNj3j7TgEHRfks4Zozc8WZd8kVa1PMkwjaJHMMLAMRv9lBBKnCzsiXZiTePdSP7A//xdJQALd3X6hwxj3JCfRV7wI/1Nnyd5nYTwOFlxzW9xfoa0QLr6z0GUEby2/lgoPI+sbF2wyGFZPBSBVKJJwyw0TZ1ZIDd2iX7URDwVplUmJvUglkbcoCpLgUiidzduE30OnhRxMf+YTHUVkRObteIBAndzRgIt3bJCzp8TgDkB8cGOUb9kCC5JxbFLHjpNC7RYVnrwTcwdNqN9k7iihGa5Xd6eajF1/Dh9xMGLbDBSp3kVz3pJqVOUMDQKxR4VKPjb7wqcpDv8+msmL6b+Nw8auibJaHGHWquacbmmnxQ8kDPDZ7TjToRj1JajFNVRdZ8Edr0BS6G1ZxAITUmePTKEMRhdVVq4o w1eLHRVO PTBwEVR9C5GqBGSpjG4vOn72bQkpbFMr7Aj5ApUzwyT/YXYJYUfpzccIxkTU6ZjmdSbGw32Yo/X0iZAe4QF3M8Ug8DD5O/1aD1jNViJ//7P7knbNCHEnz8eSlSbI0RrN8PG7C3bTKWp40cbNSbD2heePE2CtdICg08xA98NSPKUyldffdc86Zv142GK5kq/DYvz2Lkjd3+Py8cEk7UUWnPnviPV5/QIFiWDnnJKgU23ngx4tFcEbWa0SOLucVktwTsJ8H/4i96g8TUtfUejIEw7w5KMFwslSX7GC68VbaXxgIotz6k0r8sPbRLF8ix2qy0g7C X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We may hit a situation wherein we have a larger folio mapped. It is incorrect to go ahead with the collapse since some pages will be unmapped, leading to the entire folio getting unmapped. Therefore, skip the corresponding range. Signed-off-by: Dev Jain --- In the future, if at all it is required that at some point we want all the folios in the system to be of a specific order, we may split these larger folios. mm/khugepaged.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 8040b130e677..47e7c476b893 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -33,6 +33,7 @@ enum scan_result { SCAN_PMD_NULL, SCAN_PMD_NONE, SCAN_PMD_MAPPED, + SCAN_PTE_MAPPED, SCAN_EXCEED_NONE_PTE, SCAN_EXCEED_SWAP_PTE, SCAN_EXCEED_SHARED_PTE, @@ -609,6 +610,11 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma, folio = page_folio(page); VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); + if (order !=HPAGE_PMD_ORDER && folio_order(folio) >= order) { + result = SCAN_PTE_MAPPED; + goto out; + } + /* See hpage_collapse_scan_ptes(). */ if (folio_likely_mapped_shared(folio)) { ++shared; @@ -1369,6 +1375,7 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, unsigned long orders; pte_t *pte, *_pte; spinlock_t *ptl; + int found_order; pmd_t *pmd; int order; @@ -1467,6 +1474,24 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, goto out_unmap; } + found_order = folio_order(folio); + + /* + * No point of scanning. Two options: if this folio was hit + * somewhere in the middle of the scan, then drop down the + * order. Or, completely skip till the end of this folio. The + * latter gives us a higher order to start with, with atmost + * 1 << order PTEs not collapsed; the former may force us + * to end up going below order 2 and exiting. + */ + if (order != HPAGE_PMD_ORDER && found_order >= order) { + result = SCAN_PTE_MAPPED; + _address += (PAGE_SIZE << found_order); + _pte += (1UL << found_order); + pte_unmap_unlock(pte, ptl); + goto decide_order; + } + /* * We treat a single page as shared if any part of the THP * is shared. "False negatives" from @@ -1550,6 +1575,10 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, if (_address == org_address + (PAGE_SIZE << HPAGE_PMD_ORDER)) goto out; } + /* A larger folio was mapped; it will be skipped in next iteration */ + if (result == SCAN_PTE_MAPPED) + goto decide_order; + if (result != SCAN_SUCCEED) { /* Go to the next order. */ @@ -1558,6 +1587,8 @@ static int hpage_collapse_scan_ptes(struct mm_struct *mm, goto out; goto maybe_mmap_lock; } else { + +decide_order: address = _address; pte = _pte; From patchwork Mon Dec 16 16:51:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD1F2E7717F for ; Mon, 16 Dec 2024 16:53:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 362746B00BC; Mon, 16 Dec 2024 11:53:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3134F6B00C0; Mon, 16 Dec 2024 11:53:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 18A626B00C7; Mon, 16 Dec 2024 11:53:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E959C6B00BC for ; Mon, 16 Dec 2024 11:53:40 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id ABA151A0195 for ; Mon, 16 Dec 2024 16:53:40 +0000 (UTC) X-FDA: 82901418054.04.8DCC7B9 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id E996E140004 for ; Mon, 16 Dec 2024 16:53:18 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367988; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WZ4O5Num19mBVok4b6QGM7XTbrkse/VcnZ42Aznf+Nc=; b=vI82gsoULkSOXcK6atCHNQqF84Clrxf929fNKoZ0FMs5fcoMGlacI4C8L4oJG58ErCG0sL /0b3vKYhK5tLIv6dHc7vnRwoWbC1VgUz9XgSjqu52B/yJzjYPB3Zm0zl5T9Kx0Zk7loeSB eH3m6rfVy5k8sMYdnVEPKRazgpGEdmA= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; spf=pass (imf09.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367988; a=rsa-sha256; cv=none; b=4eL5SI9jugX7qH7wI+En4H2Z2SAZuhtSO2tPxKFCrc9Hx9VHOZR3lNOHqc9FNYZj5bPHcB jzd1rwCdOg3uXdNr4XkPnCIm7NM0Y/oyxvp58+aJiMONcx3uT5KloQZufUx4CoodlQ+5Xt 2y6K1KtPr7ZlXuSKVfn/L8rQpwCXQDU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 911B616F8; Mon, 16 Dec 2024 08:54:06 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B695E3F528; Mon, 16 Dec 2024 08:53:27 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 11/12] khugepaged: Enable sysfs to control order of collapse Date: Mon, 16 Dec 2024 22:21:04 +0530 Message-Id: <20241216165105.56185-12-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: E996E140004 X-Rspam-User: X-Stat-Signature: 6b4f7k5zb6c9ugzdb3wr6xfoyahs7seo X-HE-Tag: 1734367998-807489 X-HE-Meta: U2FsdGVkX18bHSUEFpI4m3qnCrCs4TudIQk5bh6sBEOQl+uUw5YJhbnpF1An3oCWBc4LMHJLB5nSA+a0drInSZxKvIl5A1qYb035f9EQTnrzTlx52N9mhA+XaoXkcFGunebGULoLXBgq0XMeoVqYK4eWu9YxQr2hLaNTORPbkrpt564iRaXRu8cdVP9/if0pNo8ewLT4yAgBXl13Ry9R8kvTij3UD8ZWTLVV6cMKBJZaRJpQonfrzghdQmOESTYt3rOGhphQGWK6Oh4qrI+hi5P1FEUmAcoom4ceLbmlXmQKMWTulsawPAyMInTlgZxtuuw2U8hKZce29q3cu2+h4lghl/d49M+pv3mktbLmmwZRaEuv+rP1TkFh/5mtHtSi8WV0NhYzI2oCF3KIzSN/WeANyKccHAiqhzbSi7n+UclHFu9C+yIqpNAH/2WT4pei2xcoFZSx4cFi/APa0ozFxzYFarOScqJsP6mgyc/ZTtqfQNBlRu34PpjHHAphyPbF08znbf0MyayU52VcBm6MuI311r5D1bx5MgUrVzfj6nGSc9p9QG/r+ZeMdsJx31piaOfCjOnr2kBmouoLRacPNWBPyP09fcuEuLDAkK3IMy8pi/vt3M6Tl0UUd0Oo/kNKkTLbErb+TKiApXEt2wtoTCHBCGC6SKMi4H9SskIv6Nqg1WWEOqX+N0pSQTvIgff75RaK2/wMB8+tAA5PD3HJwrCZE8aSbdHWQU3SFLf6RzfWrkxImU7o4bYQIRi4juI+dgqoAZ2ZMnM+5GEEYOq00I56CLZ5vz+oqfJRnM8GOhJ6Iiw/ZvkG5oU5iwVE+3igNe4/k5Pu2Wx5IyIV3hkDiHttURBu11Jv1UNO1ZbiHvtlKFmym5Eheu+m+9DwmIZXeh/Xkl42GyJPTleefe5eeVrFoS/wPm3a7sf/DYBmaOKgsV6TIHof9ZtjCM4pMyQnMvtJ3Xo5FELLK/K5MJf eA/Sr8KS TiKFc1d9g/LgApObWUxOd2DKXT3WiFThEPWC1BevhEyMrWN09XdRVf0StxH3/7vXwvAjrzRp0wbOsgKgEFwNvdlSuHXv11gR+VKDWndVigszNG0IwG63woKVwLbK0zPzHOSGT41nTnfImnODnBJBrtWYyDeKE4QsRtJUCkDktBdw3DqAP69I5nNwbAmVegaOe5Cw6cVroaEqT8IcGag7XtgUYzbA+9DaE5SmSEIzt4Ct1ScK8O6LjrdH1MKWs+lcC7Qkj5xLgmvhqbKwquv/qckWgRySjhJsXL+TydmpIY0DG1pa9VGnZjDoIptOHLQ6j3+FY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Activate khugepaged for anonymous collapse even if a single order is activated. Note that, we are still scanning the VMAs only when they are PMD-aligned/sized, for ease of implementation. Signed-off-by: Dev Jain --- mm/khugepaged.c | 37 +++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 47e7c476b893..ffc4d5aef991 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -414,24 +414,20 @@ static inline int hpage_collapse_test_exit_or_disable(struct mm_struct *mm) test_bit(MMF_DISABLE_THP, &mm->flags); } -static bool hugepage_pmd_enabled(void) +static bool thp_enabled(void) { /* * We cover the anon, shmem and the file-backed case here; file-backed * hugepages, when configured in, are determined by the global control. - * Anon pmd-sized hugepages are determined by the pmd-size control. + * Anon mTHPs are determined by the per-size control. * Shmem pmd-sized hugepages are also determined by its pmd-size control, * except when the global shmem_huge is set to SHMEM_HUGE_DENY. */ if (IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && hugepage_global_enabled()) return true; - if (test_bit(PMD_ORDER, &huge_anon_orders_always)) - return true; - if (test_bit(PMD_ORDER, &huge_anon_orders_madvise)) - return true; - if (test_bit(PMD_ORDER, &huge_anon_orders_inherit) && - hugepage_global_enabled()) + if (huge_anon_orders_always || huge_anon_orders_madvise || + (huge_anon_orders_inherit && hugepage_global_enabled())) return true; if (IS_ENABLED(CONFIG_SHMEM) && shmem_hpage_pmd_enabled()) return true; @@ -474,9 +470,9 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, unsigned long vm_flags) { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && - hugepage_pmd_enabled()) { - if (thp_vma_allowable_order(vma, vm_flags, TVA_ENFORCE_SYSFS, - PMD_ORDER)) + thp_enabled()) { + if (thp_vma_allowable_orders(vma, vm_flags, TVA_ENFORCE_SYSFS, + BIT(PMD_ORDER + 1) - 1)) __khugepaged_enter(vma->vm_mm); } } @@ -2586,8 +2582,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, progress++; break; } - if (!thp_vma_allowable_order(vma, vma->vm_flags, - TVA_ENFORCE_SYSFS, PMD_ORDER)) { + if (!thp_vma_allowable_orders(vma, vma->vm_flags, + TVA_ENFORCE_SYSFS, BIT(PMD_ORDER + 1) - 1)) { skip: progress++; continue; @@ -2611,6 +2607,11 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, khugepaged_scan.address + HPAGE_PMD_SIZE > hend); if (IS_ENABLED(CONFIG_SHMEM) && vma->vm_file) { + if (!thp_vma_allowable_order(vma, vma->vm_flags, + TVA_ENFORCE_SYSFS, PMD_ORDER)) { + khugepaged_scan.address += HPAGE_PMD_SIZE; + continue; + } struct file *file = get_file(vma->vm_file); pgoff_t pgoff = linear_page_index(vma, khugepaged_scan.address); @@ -2689,7 +2690,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, static int khugepaged_has_work(void) { - return !list_empty(&khugepaged_scan.mm_head) && hugepage_pmd_enabled(); + return !list_empty(&khugepaged_scan.mm_head) && thp_enabled(); } static int khugepaged_wait_event(void) @@ -2762,7 +2763,7 @@ static void khugepaged_wait_work(void) return; } - if (hugepage_pmd_enabled()) + if (thp_enabled()) wait_event_freezable(khugepaged_wait, khugepaged_wait_event()); } @@ -2793,7 +2794,7 @@ static void set_recommended_min_free_kbytes(void) int nr_zones = 0; unsigned long recommended_min; - if (!hugepage_pmd_enabled()) { + if (!thp_enabled()) { calculate_min_free_kbytes(); goto update_wmarks; } @@ -2843,7 +2844,7 @@ int start_stop_khugepaged(void) int err = 0; mutex_lock(&khugepaged_mutex); - if (hugepage_pmd_enabled()) { + if (thp_enabled()) { if (!khugepaged_thread) khugepaged_thread = kthread_run(khugepaged, NULL, "khugepaged"); @@ -2869,7 +2870,7 @@ int start_stop_khugepaged(void) void khugepaged_min_free_kbytes_update(void) { mutex_lock(&khugepaged_mutex); - if (hugepage_pmd_enabled() && khugepaged_thread) + if (thp_enabled() && khugepaged_thread) set_recommended_min_free_kbytes(); mutex_unlock(&khugepaged_mutex); } From patchwork Mon Dec 16 16:51:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13910086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28655E77180 for ; Mon, 16 Dec 2024 16:53:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ADB946B00C9; Mon, 16 Dec 2024 11:53:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A8AE56B00C8; Mon, 16 Dec 2024 11:53:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 957A96B00C9; Mon, 16 Dec 2024 11:53:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 71E2B6B00B4 for ; Mon, 16 Dec 2024 11:53:52 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1B3AC140E73 for ; Mon, 16 Dec 2024 16:53:52 +0000 (UTC) X-FDA: 82901418180.10.38416E4 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 185D740011 for ; Mon, 16 Dec 2024 16:53:27 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734367998; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xhFPveaZorJV7RkvdDwNQ8gdAUT2oKTD6wdqzlLHLWs=; b=lANLQmQZGGytKoOlg6PIQ2ysahJ2J+2+pFVprl8Rl3Xneo++iA7In105oG/Xdu5YFvKgok LbcuEqO8iKjRZRkLvPYHU+UtZZEupwMgKviOHkz1HNmAScBcg8VCVidFjpoYCwwy7Kz+fr lAVhZkfDd1POpKgCB8/KmiENSY0N7do= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734367998; a=rsa-sha256; cv=none; b=GL6Zj6owUglzfgyGViN7AkpRfqUKnl/k1TwppqLWg+D7Mj1UYz1+icIt8AjejxuAFaO6qT ox+tpYZrgLw0qnzXwhZIELQU4cBhDUpIrJD6LjahvYgsMVIOg++2UThhkAHhME0E5o2szD pkJSMH0PBkUbN3cGaK93EYNOR14maA0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0479416F8; Mon, 16 Dec 2024 08:54:18 -0800 (PST) Received: from K4MQJ0H1H2.arm.com (unknown [10.163.78.212]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 1A31E3F528; Mon, 16 Dec 2024 08:53:38 -0800 (PST) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org, kirill.shutemov@linux.intel.com Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, catalin.marinas@arm.com, cl@gentwo.org, vbabka@suse.cz, mhocko@suse.com, apopple@nvidia.com, dave.hansen@linux.intel.com, will@kernel.org, baohua@kernel.org, jack@suse.cz, srivatsa@csail.mit.edu, haowenchao22@gmail.com, hughd@google.com, aneesh.kumar@kernel.org, yang@os.amperecomputing.com, peterx@redhat.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, ziy@nvidia.com, jglisse@google.com, surenb@google.com, vishal.moola@gmail.com, zokeefe@google.com, zhengqi.arch@bytedance.com, jhubbard@nvidia.com, 21cnbao@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dev Jain Subject: [RFC PATCH 12/12] selftests/mm: khugepaged: Enlighten for mTHP collapse Date: Mon, 16 Dec 2024 22:21:05 +0530 Message-Id: <20241216165105.56185-13-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20241216165105.56185-1-dev.jain@arm.com> References: <20241216165105.56185-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 185D740011 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: a4jnbh1b6amttaexa7x6j9oxqns8s8eb X-HE-Tag: 1734368007-108322 X-HE-Meta: U2FsdGVkX1+tnbsV56W7hwh+8M9skcnjljqVVcJ4XOVun9vDaVSZmc+gS2BXeS1iK3a4qE2EZ+OcxzHvvaHgIAIxUk8igmdyB84oIByNM5cCFDhAQRI+4enAFMh6OpsteaBfPfWDn084zP4Ee8pPH8YkHbwa+oMv5yVQAuqprbRv8wKik7yoEDVCGNVouUfy/NmRI6mQvEcC4Ms0scOCYc+FdgZvGYBTaaffW/5Siuh3JJ/FpVGYvAKDl75Q4YYbrNcqgoNfRYSc3KkNKlWYaAF3Bgo2umW88sKVNW1VxIbkyrKKtm7V0f3To04GkiOPqLQrequOnWh59IqSetxzsN0pkdmWb4wsfsMkx3XZOPqlnPPPUHjgY4aABDO47483GkLrDBysqZDZnkab/oZBMB60p6+KK0KuNt/rpjRR+axshVvej6533kP37ncx6WKIQl2ij+SoMst/S+7m4R2/3rA4uVlTCNNEm1hVv1JG2fGxap9HxHi+o3YrQ3CXpXM1Ywd32DcduerZ18iKBnL5lOWMT3xuVvjU23FsGsMwxG4xO661CbDuuVijqD7n1NEOX1CPtwLWgXLuqLGRRtVs16DCY3UpvtYxfuC+Qm1ylhGLkYUA7zJT9BlcIWyhfU/dbM5AKFLDg+mJkHQvtfqsrrMsf7hSEVsI6XsfgHFRcPf+oq7+UTd8yAcFUzrV1/QehsVSSbIA6muDlzvWlWkO7Tu9VNlmcgkELqG3Rd50h+p/UIVO1BbKDUPkFznwUuWk0fszS9yEHmDl2g779ny6mgIh/ytEbIocmYp1hs66LrBtM5f8Ek/ga7lUnS47JGJvO22cMRv4dHDyvtou+OHaF/viL6+boZ9JlvOPM6lxvGmLipSe1QzM4y5YMJ6d7lI+3FEYr5mlsp7dDaRtP5E+qTaSGW3tCGYqK3BA94uozDCEmE4XGLfh9G1FnJM/UPHQdf3OhDrn4a1v30mNOAD NASuKHuT C8R468b7MOwiNCW5zd/DM9IvwAmLBwGL0Ap7cnzN/riSqHyeDGqvK58R7JvXJDwsnTVee8e3dcQ8/3Ry0F8LqwD1yiMSeFtaV0burmzA0gk6HiyffG57VGXOxON9iR4Jv3Oa3t9Z7+CW+uPV9QNvXLuACEUooFgK8QO3KuBt5GugfnBQbESIGXO3jEJN03a0RLM9z8Qre5TPTngzQydWw2mWcCSAxLS/T/6WCGZndUadyT1q4s51+VR3KIAS4th2KA5tIo4shM6tWiKVJxUHFrNdwcjlyCYFZPJAzY8aYD3a+9q7A/CaTlO12BqG+Ld4esst4 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: One of the testcases triggers a CoW on the 255th page (0-indexing) with max_ptes_shared = 256. This leads to 0-254 pages (255 in number) being unshared, and 257 pages shared, exceeding the constraint. Suppose we run the test as ./khugepaged -s 2. Therefore, khugepaged starts collapsing the range to order-2 folios, since PMD-collapse will fail due to the constraint. When the scan reaches 254-257 PTE range, because at least one PTE in this range is writable, with other 3 being read-only, khugepaged collapses this into an order-2 mTHP, resulting in 3 extra PTEs getting unshared. After this, we encounter a 4-sized chunk of read-only PTEs, and mTHP collapse stops according to the scaled constraint, but the number of shared PTEs have now come under the constraint for PMD-sized THPs. Therefore, the next scan of khugepaged will be able to collapse this range into a PMD-mapped hugepage, leading to failure of this subtest. Fix this by reducing the CoW range. Note: The only objective of this patch is to make the test work for the PMD-case; no extension has been made for testing for mTHPs. Signed-off-by: Dev Jain --- tools/testing/selftests/mm/khugepaged.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/mm/khugepaged.c b/tools/testing/selftests/mm/khugepaged.c index 8a4d34cce36b..143c4ad9f6a1 100644 --- a/tools/testing/selftests/mm/khugepaged.c +++ b/tools/testing/selftests/mm/khugepaged.c @@ -981,6 +981,7 @@ static void collapse_fork_compound(struct collapse_context *c, struct mem_ops *o static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops *ops) { int max_ptes_shared = thp_read_num("khugepaged/max_ptes_shared"); + int fault_nr_pages = is_anon(ops) ? 1 << anon_order : 1; int wstatus; void *p; @@ -997,8 +998,8 @@ static void collapse_max_ptes_shared(struct collapse_context *c, struct mem_ops fail("Fail"); printf("Trigger CoW on page %d of %d...", - hpage_pmd_nr - max_ptes_shared - 1, hpage_pmd_nr); - ops->fault(p, 0, (hpage_pmd_nr - max_ptes_shared - 1) * page_size); + hpage_pmd_nr - max_ptes_shared - fault_nr_pages, hpage_pmd_nr); + ops->fault(p, 0, (hpage_pmd_nr - max_ptes_shared - fault_nr_pages) * page_size); if (ops->check_huge(p, 0)) success("OK"); else