From patchwork Mon Jan 29 05:45:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lance Yang X-Patchwork-Id: 13534928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 696FFC47422 for ; Mon, 29 Jan 2024 05:46:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9DA316B0075; Mon, 29 Jan 2024 00:46:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 989A56B0078; Mon, 29 Jan 2024 00:46:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8512C6B007B; Mon, 29 Jan 2024 00:46:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 76F996B0075 for ; Mon, 29 Jan 2024 00:46:18 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EAEA21201F5 for ; Mon, 29 Jan 2024 05:46:17 +0000 (UTC) X-FDA: 81731263194.23.1229232 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf06.hostedemail.com (Postfix) with ESMTP id 3EEF5180004 for ; Mon, 29 Jan 2024 05:46:16 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AoB1qppN; spf=pass (imf06.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706507176; a=rsa-sha256; cv=none; b=mJIJQH/xdSHKdIO0JseSuNajGR8KVdQ87gVnPCbJgLFLj0HJtg1I2qkP8dSWhQMbWBOry9 /AUfbUsWNf/9EzUCnJSz86Q+bFPfzhKek0gtQhJZtCr/RGcGcuDtUL21lpC/cpihsX/pud jgdI74HAzFcQ6COYssAQ75+B8RciD1U= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AoB1qppN; spf=pass (imf06.hostedemail.com: domain of ioworker0@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=ioworker0@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706507176; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=IHuIHgsq0pkJV9MVOB176xiitDstTf9e4RykLNwMz7o=; b=jEFTAUwnlmIFgDbZuZdgSCQoeXFWC1hORwQBR2KuBzK+mxUexu/T9cTH/LyZaTB9fI8wAh 2bgxiLtLC0Q2VlJpVL29F8JwZs1X1JW/n6BXsSZ5spjDhEAzXxNA5rEr1Mt8ViUfFiXR0I wWmYs4Ulmb7ai/GA9ZzEepZPGNHsWMU= Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-1d8e78270c1so2129765ad.0 for ; Sun, 28 Jan 2024 21:46:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1706507175; x=1707111975; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=IHuIHgsq0pkJV9MVOB176xiitDstTf9e4RykLNwMz7o=; b=AoB1qppNxgf2x6krVNQ1cBt+eXi6TCyCGToShB8sBUP4+QJHufKoEn2uikl1ZVGSCi 7rDXveQHr9sFLzKTZIWlhM34d6DHIcyAnZgeJM57ih7mWBQg95T67R3ynu8f9550JrkC PCGaRgrhGYW0kv2kKpQiLms5zxDFedo6xy/vyWdY41paij2OENbE4eDV9dgebjMJH8AU J66m3AqQx2dz4328GF4e44lqPoBbBGKL59A0sgsh3BsvWC7BTdrHquCUoN5FfeFVcAKP FGXUcrF2nrR0RJxKU4DLqtgwNKZbFjSJQMNjalaRoTGhCGtG7WI5Cf8Djn7cA/X/z1SO JZjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706507175; x=1707111975; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IHuIHgsq0pkJV9MVOB176xiitDstTf9e4RykLNwMz7o=; b=U+rQl3w/c3JxOOM2yToyXDshFqnbq8q8qUY0tcViHEOx0CskEFGE+iK8nL5RpuZdHJ 2d/6S2fdEkQDzMiZ0eG2KO9GHbTtuT4tfwIfVLBTDIQNm2/X2C/2f8tc1RDXhy9K42zL PW0ZZ5QuM6C64pItrP9S58K1OJVhMdgnrrDlkVZPtJ5nEEnN2QdSz7aUY/Y/CBJa0G0D 09ldpr9XAP7BvhfRtWEE5RmXnyPfT7t5349S1gEWlWnNoPRjbjaDx+JH5JZmp+159MaA aKoK7swlApx82TpidOCSE70DZB3kwrkcOxtZMcfvWkdgFEgMJOp70IzHoe5yjKOWrAuX lxfA== X-Gm-Message-State: AOJu0YyyAv41xUqPmv2dTP3bN6/RpPJ984pM6wuT6J/AN0QltG7UCkzM nqC7vSHoZ7h2OEGJBN29o0hL5cWlb3H5CM/prAZQgqwRfLQici+R X-Google-Smtp-Source: AGHT+IEOZOP87PczPRaEtLnjXj7L60nqKEcW0EEF0dmvYeE8VsnLp8N+eQ3vn5EbGO/n6iJ6WNs+WQ== X-Received: by 2002:a17:902:ec06:b0:1d8:f3c7:fb99 with SMTP id l6-20020a170902ec0600b001d8f3c7fb99mr102478pld.28.1706507174938; Sun, 28 Jan 2024 21:46:14 -0800 (PST) Received: from LancedeMBP.lan ([112.10.240.49]) by smtp.gmail.com with ESMTPSA id n18-20020a170902d2d200b001d6eed31a1bsm4453110plc.245.2024.01.28.21.46.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 Jan 2024 21:46:14 -0800 (PST) From: Lance Yang To: akpm@linux-foundation.org Cc: mhocko@suse.com, zokeefe@google.com, david@redhat.com, songmuchun@bytedance.com, shy828301@gmail.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH 1/1] mm/khugepaged: bypassing unnecessary scans with MMF_DISABLE_THP check Date: Mon, 29 Jan 2024 13:45:51 +0800 Message-Id: <20240129054551.57728-1-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 3EEF5180004 X-Stat-Signature: 9upeorgrpp6mf7epd3bw3gorcfckk1mi X-Rspam-User: X-HE-Tag: 1706507176-13955 X-HE-Meta: U2FsdGVkX1/BLJA6wA9gmMLTA6IJ+PVJ923y5QSLBuB2NqV4s7bowLx/Wf0ZjTjRSYz+eooLzi3x6jhUu1jKnU8WEAk4IwKnWr+1NTu8DhSiX6dLLebIL6CHKVlFMaqqElOLrGMXx55ZVYaA1HdYu0AnxCRHIgNdk8LE8oIceQeSia6AtXdnsPc/PkZ4mTm6tG4Bts1qD8R0keCoHs9OyejO5h8S/xhDzzp94goCxwI1yNYFU9Blf0ziQYU/OUKMc363tfAOE+r2tL1edhE17C7oIKFfq914QRXfIVtWaqO2GKsW0E8Z3/IOdR7AuXP5oelIvCy7LyNWilks0QcDhMee1/n3wgGpG8mLo4GE0aSduKnPk4VsnfLUhH0BV/8kbHeh4ljlK0HqkSV/txjLoKCOGGnsCHayflDpd5sxjBb7vx0mYXM4ekNC871Qobu58tQX7CXU51LpxlKvSOJLw+LJQLfxiSdRrCP3L4GCSJVhnXCUIBnVblp086q94YMISlNDqnwmCFxTiFBbNvxrjVQPBgp5+yrrD7afM5Vtm81GGQoRfH39yGDHXElAfn3yI9PQlr4RtCXcrkOzaneJZB55wwK3VtV3ERv2xxs6QXuCCFYcB3YWjrduBM/D4IivG9s3A6G4dJEfNgERoKztO1O/OtPeip9BfK5waeA+TEwzlDWqCDd4rI2DTFenL8bJoFdVSVxRP2KgS+7nQUZLL8Yk24cIwFrxoeQlx3guiT5ZDxAZIxCo/TwvUojRl2IYE+ua4wcj/nOQT8WyEm0NIQkMNlBDOlhr548g24MZXd8Cl4Ury+4FBWItQEeOoMygd1oWosgLcYl2gTji6NO6qRtP3mXDvJL52lRtRlHlf6ETuLMBHfh8cfz0DP5N2se9WeWnHjgsm0SJAxQns2pdOdYm5yuOF0AldPJ8ZAjdFX088epZ6IGGoTDLfrTCQlNC2p9Dze4gB8bvBqluvdP hXdEJHZA hjfSrB7SM7VvMcrDYDZuqZaghIvFux/tnOo31UzDk4b2sYwZQxcqX3pi/JVMZncyzC2mV4xN5eXtZs6QJKALU7iXlbvRLOGreqxdI+ErU+gyRrMrn1BLa+awwgHka2Rgrg82zHWvNr07b5vCVMp4aHkQzKo7XcVObIRMzzW/QmJf17f+vuo1e3aNlv5sndC+eyCyxJk7admXu6wSeEo8+IQb7SLFulO0MCJhwoGM/WSo1ZWIX1nNg+6gA7rT5WBMeLQTdhZILZ2OLaBvSzbwQCCF6PXZhzJmMo4BNfeJxkIvEUkr2gymlCwX/2HPkdhStr480m/Zc35WahSOer91NdEBRVjC7ed4DRv4j5I5cwzmTDDWGBdM4juH6ZY+/0k/8StKxZjDlZdNhoXP+od+0wo9ht3rTyV7JSRzXFy0q56IEf5jBU6vADqJlIFePasrNPSqYWklTEFjnLUFkam3zxH4zuXK/BraQvgdtlbzyQDmUZdE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: khugepaged scans the entire address space in the background for each given mm, looking for opportunities to merge sequences of basic pages into huge pages. However, when an mm is inserted to the mm_slots list, and the MMF_DISABLE_THP flag is set later, this scanning process becomes unnecessary for that mm and can be skipped to avoid redundant operations, especially in scenarios with a large address space. This commit introduces a check before each scanning process to test the MMF_DISABLE_THP flag for the given mm; if the flag is set, the scanning process is bypassed, thereby improving the efficiency of khugepaged. Signed-off-by: Lance Yang Acked-by: David Hildenbrand --- mm/khugepaged.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2b219acb528e..d6a700834edc 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -410,6 +410,12 @@ static inline int hpage_collapse_test_exit(struct mm_struct *mm) return atomic_read(&mm->mm_users) == 0; } +static inline int hpage_collapse_test_exit_or_disable(struct mm_struct *mm) +{ + return hpage_collapse_test_exit(mm) || + test_bit(MMF_DISABLE_THP, &mm->flags); +} + void __khugepaged_enter(struct mm_struct *mm) { struct khugepaged_mm_slot *mm_slot; @@ -1422,7 +1428,7 @@ static void collect_mm_slot(struct khugepaged_mm_slot *mm_slot) lockdep_assert_held(&khugepaged_mm_lock); - if (hpage_collapse_test_exit(mm)) { + if (hpage_collapse_test_exit_or_disable(mm)) { /* free mm_slot */ hash_del(&slot->hash); list_del(&slot->mm_node); @@ -2360,7 +2366,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, goto breakouterloop_mmap_lock; progress++; - if (unlikely(hpage_collapse_test_exit(mm))) + if (unlikely(hpage_collapse_test_exit_or_disable(mm))) goto breakouterloop; vma_iter_init(&vmi, mm, khugepaged_scan.address); @@ -2368,7 +2374,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, unsigned long hstart, hend; cond_resched(); - if (unlikely(hpage_collapse_test_exit(mm))) { + if (unlikely(hpage_collapse_test_exit_or_disable(mm))) { progress++; break; } @@ -2390,7 +2396,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, bool mmap_locked = true; cond_resched(); - if (unlikely(hpage_collapse_test_exit(mm))) + if (unlikely(hpage_collapse_test_exit_or_disable(mm))) goto breakouterloop; VM_BUG_ON(khugepaged_scan.address < hstart || @@ -2408,7 +2414,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, fput(file); if (*result == SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); - if (hpage_collapse_test_exit(mm)) + if (hpage_collapse_test_exit_or_disable(mm)) goto breakouterloop; *result = collapse_pte_mapped_thp(mm, khugepaged_scan.address, false); @@ -2450,7 +2456,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, * Release the current mm_slot if this mm is about to die, or * if we scanned all vmas of this mm. */ - if (hpage_collapse_test_exit(mm) || !vma) { + if (hpage_collapse_test_exit_or_disable(mm) || !vma) { /* * Make sure that if mm_users is reaching zero while * khugepaged runs here, khugepaged_exit will find