From patchwork Mon Feb 17 14:08:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13977914 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C6AAC021AA for ; Mon, 17 Feb 2025 14:08:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 31D14280061; Mon, 17 Feb 2025 09:08:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2CDC0280059; Mon, 17 Feb 2025 09:08:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 146E9280061; Mon, 17 Feb 2025 09:08:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EA559280059 for ; Mon, 17 Feb 2025 09:08:45 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A0A61B04EC for ; Mon, 17 Feb 2025 14:08:45 +0000 (UTC) X-FDA: 83129617410.12.F99C290 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf02.hostedemail.com (Postfix) with ESMTP id D80E480004 for ; Mon, 17 Feb 2025 14:08:43 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739801324; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MNbungrrFJz+FPgf+VcHoNwc+UNtCM9gplHqa2WAA0I=; b=O7vjVN5nsXzkS4ucC41kQQoxbZsAqbKHJE4gAhm0U8IHf8Q7ui5eN0577+uC0+kvY6/Xbj mMMryjB8e4wBySCECmkop7XaE5Xlt/BsBuG0WZyq2uPg452zEiKzy6vsDtHA//x3hwBeoH u9my/omh8hixvYcNZlKhDlIGbBsPm2I= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739801324; a=rsa-sha256; cv=none; b=Tj+2LHe9lnRIBekgWsyWKsg/E6/Igu++9XiQsITBA04OJlurs/opB6cwAyQOwQfALWe+W2 +zQQrlu9sRUV7n7eTSUpcpq5wy8tH4VxOnfp3kmw5z8QUdYjprxR1+L96YPF+XGGm3nuue BrzMdEGCVbBbJYdwMiAlUJdDTdcr88k= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6D5B91A2D; Mon, 17 Feb 2025 06:09:02 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id F384B3F6A8; Mon, 17 Feb 2025 06:08:40 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , David Hildenbrand , "Matthew Wilcox (Oracle)" , Mark Rutland , Anshuman Khandual , Alexandre Ghiti , Kevin Brodsky Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 09/14] mm/vmalloc: Gracefully unmap huge ptes Date: Mon, 17 Feb 2025 14:08:01 +0000 Message-ID: <20250217140809.1702789-10-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250217140809.1702789-1-ryan.roberts@arm.com> References: <20250217140809.1702789-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D80E480004 X-Stat-Signature: tuc6emw9jdrm8jdusswqjb4ceabw9uir X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1739801323-844116 X-HE-Meta: U2FsdGVkX1/jh/rT+VdzLBgfnVtanbVkwAA4HWanEgvSOpirCrRptlCHpMwnKZ+xp1UaC7Ko7PfsBmGrjfZoNtYt9GRJXse0eIHBmzazhTf/9+Vc0XSqRI1AcX3VUYKpI6gk8nIai3z8SV9mn4GYb06qswajzTPJA4qCKXWPMQeYM93dLV1CjNvK8g8gw41L4ZUjADYJdbV/+JyZyLJMpmlNu2Y0zFE8f9SWNo22M4zZk5O7FP7rZRS+clVR4pZi7+jfz+VMI2Qtdwavv8niVICYu2Dgc3+jkv9oEctkOyRFslX00eq0Ua0HgNT6O7/F+rQI5r7jnSFu9NUAjC2qW+hGEjGNkK2ayKaxxmuK7ZJwgH5Xvtv8VpiThcRgMklTrbjkQKajbkPPANBQYtfy8zxjgFPCTNQxzuwrpxNI+c9K9puLCGIVg3m279LAwWBcodOiXd29Jvyqzrl6aUqrTuK0Wn2gM1jCih3VRaG7BbkFX9QSrpyhIf7sDdMEtcAWDHVj5bBwWrVSkxREGyN+lLTeWocn8Ik5j5SGiuTgLA+Mlgkt9AV+kJZRcRKh3tKOf761LXO8oymDp+WZcZ4r2cMOLyKqdPy9fy09rWsXepYZ0pKy25yE5RumnH3aiYxl2mAnA4omeuGqNzW5JAHnu0emVtgMqKCy9jQDCkMOtnrBUy4gQOkpctyQhn/kimjiVIPa7kxu7eMiJmloklrXBvweH8SYMBH7ZzU5+n99jOrgsanktEERT21RophVhrGoiduP5RWz1Mqfppwp1rPVof5co0wd+FS2ndXN+uGGmJB1shYssIbeplwnGMV3uPyNnAyj6b5QoAhh1vP9mL2pbSPUqZx+mrZXcvTNdSLWMBtgIn+IL3kgeuwep2b34euw/pmg4Ti8pi/cYQcZ1ac1kvBfBXb57ndkODSk7Ym5zGeiK6z+8I8iRDaujmoW1LARDCV8oA8y0vK9XW6lSYd cTGF0Xq6 cTPtIkRVlGo5JVGYjZXM0XIYyQHM5mn1T9w92up5Mdkazvp1uajarFIL1iZMwF2Dw2VuvetM0kXabQget+Zm/MB3YgLewgdSO6UwUzIGYv3/I3CHmpZoKJI6oSkAia7/Q2xspXTpw9oXY0na9SuLOe5YbFaeNmra+boHrUGyfnu2VjO6CB3hivsOpOR7wn2T/cITO X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit f7ee1f13d606 ("mm/vmalloc: enable mapping of huge pages at pte level in vmap") added its support by reusing the set_huge_pte_at() API, which is otherwise only used for user mappings. But when unmapping those huge ptes, it continued to call ptep_get_and_clear(), which is a layering violation. To date, the only arch to implement this support is powerpc and it all happens to work ok for it. But arm64's implementation of ptep_get_and_clear() can not be safely used to clear a previous set_huge_pte_at(). So let's introduce a new arch opt-in function, arch_vmap_pte_range_unmap_size(), which can provide the size of a (present) pte. Then we can call huge_ptep_get_and_clear() to tear it down properly. Note that if vunmap_range() is called with a range that starts in the middle of a huge pte-mapped page, we must unmap the entire huge page so the behaviour is consistent with pmd and pud block mappings. In this case emit a warning just like we do for pmd/pud mappings. Reviewed-by: Anshuman Khandual Signed-off-by: Ryan Roberts Reviewed-by: Uladzislau Rezki (Sony) --- include/linux/vmalloc.h | 8 ++++++++ mm/vmalloc.c | 18 ++++++++++++++++-- 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 31e9ffd936e3..16dd4cba64f2 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -113,6 +113,14 @@ static inline unsigned long arch_vmap_pte_range_map_size(unsigned long addr, uns } #endif +#ifndef arch_vmap_pte_range_unmap_size +static inline unsigned long arch_vmap_pte_range_unmap_size(unsigned long addr, + pte_t *ptep) +{ + return PAGE_SIZE; +} +#endif + #ifndef arch_vmap_pte_supported_shift static inline int arch_vmap_pte_supported_shift(unsigned long size) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a7e34e6936d2..68950b1824d0 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -350,12 +350,26 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) { pte_t *pte; + pte_t ptent; + unsigned long size = PAGE_SIZE; pte = pte_offset_kernel(pmd, addr); do { - pte_t ptent = ptep_get_and_clear(&init_mm, addr, pte); +#ifdef CONFIG_HUGETLB_PAGE + size = arch_vmap_pte_range_unmap_size(addr, pte); + if (size != PAGE_SIZE) { + if (WARN_ON(!IS_ALIGNED(addr, size))) { + addr = ALIGN_DOWN(addr, size); + pte = PTR_ALIGN_DOWN(pte, sizeof(*pte) * (size >> PAGE_SHIFT)); + } + ptent = huge_ptep_get_and_clear(&init_mm, addr, pte, size); + if (WARN_ON(end - addr < size)) + size = end - addr; + } else +#endif + ptent = ptep_get_and_clear(&init_mm, addr, pte); WARN_ON(!pte_none(ptent) && !pte_present(ptent)); - } while (pte++, addr += PAGE_SIZE, addr != end); + } while (pte += (size >> PAGE_SHIFT), addr += size, addr != end); *mask |= PGTBL_PTE_MODIFIED; }