From patchwork Mon Mar 3 14:15:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13998904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D55BC282D1 for ; Mon, 3 Mar 2025 14:16:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A50026B0089; Mon, 3 Mar 2025 09:16:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D8E96B008A; Mon, 3 Mar 2025 09:16:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 878836B008C; Mon, 3 Mar 2025 09:16:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 66F3B6B0089 for ; Mon, 3 Mar 2025 09:16:02 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 12C3B1C7E1E for ; Mon, 3 Mar 2025 14:16:02 +0000 (UTC) X-FDA: 83180438964.04.00CCC95 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 241E740035 for ; Mon, 3 Mar 2025 14:15:59 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741011360; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4BoBgGHAPTj4pbFzwDuoxZ0IChG5D4eAVKUd6vh7p88=; b=NRF9hgPaUZefn1gTQnHuccaqmdJXETArUbV4IbHkqJ55d873+7pdE9tuOF4vcu3qZ+58oQ rC6Tx8A/U6COqSq4vKhWKGF7IeKMEYaSIUM+QFLuQNZhKUWG0f8I4u1aBd1OlqSTVXjQLl J931Q399eTeZwAxUovII/ayQOaqS5vA= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741011360; a=rsa-sha256; cv=none; b=sSWSq5Hfe2J05OxY6U9PmqXO6nWppnJ15NaJQw78/11B4L3+L9HknRicTmt8Z4J3CSa3nl IpXHVF7q3A3C8P2Gmd/6kPfEjzXTbWz+m2Ad9KG4McMq5gRCeQE2OAl0UNd1Ccd3kzVMbc yKgUmcSvwXMJcqVee8x/on/5QRqsdOo= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1ABF8175A; Mon, 3 Mar 2025 06:16:13 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EB6783F66E; Mon, 3 Mar 2025 06:15:56 -0800 (PST) From: Ryan Roberts To: Andrew Morton , "David S. Miller" , Andreas Larsson , Juergen Gross , Boris Ostrovsky , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , "Matthew Wilcox (Oracle)" , Catalin Marinas Cc: Ryan Roberts , linux-mm@kvack.org, sparclinux@vger.kernel.org, xen-devel@lists.xenproject.org, linux-kernel@vger.kernel.org, David Hildenbrand Subject: [PATCH v2 2/5] fs/proc/task_mmu: Reduce scope of lazy mmu region Date: Mon, 3 Mar 2025 14:15:36 +0000 Message-ID: <20250303141542.3371656-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250303141542.3371656-1-ryan.roberts@arm.com> References: <20250303141542.3371656-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: tq7tf8mjd3dot53y18i6cqyqfrdfrefo X-Rspamd-Queue-Id: 241E740035 X-Rspamd-Server: rspam06 X-Rspam-User: X-HE-Tag: 1741011359-583186 X-HE-Meta: U2FsdGVkX1/FHkzP7B4YYh0Y20S4Boy1qj4OYq+cFBkt3STNqNloEt/SZfjfTPcxQHyTyg7qHSdq60mnITBkD65IVtV+IKgL/vPkbgmUyIitRTDyNvHnW3jPQzCJUHU9IB8ddMrKqs12+fYEmBLPwzmpLzJbvJXrseWGXABzqphGdvNhL/kucDOLvFCP+YtRfa6+n36D2VLKEkyXM0XT2k42lerrWcEmB8evq41PpI2ULWDlsVEZYwbI8ltCI2t5XLTbH18ZRy0qw1aCfwAaiWdKsox5RfzrDrD1KyIDkWxwQNQ2dxiGI7OmTI6NpRiVEI+UZ8fbAFYSemSuL5Az95r+bMoWLgIe+UkEO3v05usM5S1h5R+Cx3GQwQD7Wpbt/9qWhMLEx3I7t7gvUCuiDxPMoRhI3Y7RW1eOeOk4rPyao4LWZ/9ih8uqo/ueymu91oxI//3xrJLSOU5burFiyItDMlg+jy5RXUvOlP7YI6eEWntPA8edeOMc1ReoVX4Zf54gGrUvmGam6DGew5rS6NuPJ7clDlDoOHV8I03lOhKJVRmRDc5ADztz+1YZoW3IXIy3QD2VAoqOg/SrjqnpbBHyzP3OiD6O3svXtjNsY7HoBjfHk0UFHgkQV2t7Ou3Co5lhWOA72Ee9Owph/Va/aj8HYVWhQQ9e7b3Qf3TcapqA2sbFgn32PZHff9C5/msYVgiEoGLo6xqz/iS6W8Nledx7WPTqCEX/SEiCUzDlMXBBFxNc2sTV/FO3LM7uq6ubUS37NVuaMVEE7gncmAKBPX6qeVI4vbpaQUuXjIhw2i8j2n7B3LiWqy5OeeYqdygAIf/zXDmA1esTczy/ShK4d0m463G2D80lYC3E0t0qdI4MrfpWycm9IVIMAeydXQe6xaOiFi66JZAl5YlRQjK7MstksghHW9JSKUu0AYHxsgiEmO+pu6J1P4P8HKM5EUy6tTUFVFaYx7mtiCzEoKo ox/ImqMe +jlFzqxGR+6esRJkfKHPnqC9WhJS21DuMhzNRvt6HJBO7lNfW86+QuFs57B5bWyniMz4KDhEBpIqUIIKwCjPHM+p0tEYXMeyw99lyZUewN+eY+TAw4OzifbWQX+/7fOVf38PrunlTnezmiF243aBzCXrsLjkIBih7gAyl9j13VGuaPYx0nFDfYCeZtm1IHP1MfPoGGYD/Ugx3JG6BLw9NpgcNve74AMvazf/w X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Update the way arch_[enter|leave]_lazy_mmu_mode() is called in pagemap_scan_pmd_entry() to follow the normal pattern of holding the ptl for user space mappings. As a result the scope is reduced to only the pte table, but that's where most of the performance win is. While I believe there wasn't technically a bug here, the original scope made it easier to accidentally nest or, worse, accidentally call something like kmap() which would expect an immediate mode pte modification but it would end up deferred. Acked-by: David Hildenbrand Signed-off-by: Ryan Roberts --- fs/proc/task_mmu.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index c17615e21a5d..b0f189815512 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -2459,22 +2459,19 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, spinlock_t *ptl; int ret; - arch_enter_lazy_mmu_mode(); - ret = pagemap_scan_thp_entry(pmd, start, end, walk); - if (ret != -ENOENT) { - arch_leave_lazy_mmu_mode(); + if (ret != -ENOENT) return ret; - } ret = 0; start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl); if (!pte) { - arch_leave_lazy_mmu_mode(); walk->action = ACTION_AGAIN; return 0; } + arch_enter_lazy_mmu_mode(); + if ((p->arg.flags & PM_SCAN_WP_MATCHING) && !p->vec_out) { /* Fast path for performing exclusive WP */ for (addr = start; addr != end; pte++, addr += PAGE_SIZE) { @@ -2543,8 +2540,8 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, if (flush_end) flush_tlb_range(vma, start, addr); - pte_unmap_unlock(start_pte, ptl); arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); cond_resched(); return ret;