From patchwork Mon Oct 14 10:58:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13834673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2C30D1A45A for ; Mon, 14 Oct 2024 10:59:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 557316B0082; Mon, 14 Oct 2024 06:59:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5074A6B0083; Mon, 14 Oct 2024 06:59:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F6C76B008C; Mon, 14 Oct 2024 06:59:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 212826B0082 for ; Mon, 14 Oct 2024 06:59:44 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id D599B140D1B for ; Mon, 14 Oct 2024 10:59:35 +0000 (UTC) X-FDA: 82671912120.16.9F74464 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id 170DAC0009 for ; Mon, 14 Oct 2024 10:59:35 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1728903509; a=rsa-sha256; cv=none; b=zaad3BYU6OMGCOfrYXXrwwXAhqlJFLGtCqs2FFsS/wbamxRIDmyltHaylG0JiV33FJrV/Y edv559BDwCmD5r5xLGcTv150st6KQRRJNtvuyPZzvtoqkz5ssjaApqdX+eQyOQeZH4u00W ZleGXfiLab0FaUKmwiNQ85YkyZ4ikms= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1728903509; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qPt13oBtsQSK+UGOOEWOc/EwkFI9q79bukNoe+ZJFuA=; b=C29wg4eNLloR/QrJvrGCt+hZhvKO0bAcHB/ORgk9PKuxrfggr4Yu462rNt4gL9I3HaBLAq t3doBdD5gnA+YaHrh9vFngv5uOvGuni2kGg9DhBJJZFodObPmvK2PInZ10W+doraxRBzip lFGRF2rhMAsZYknZphsmhoEVxTT2Kb8= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E3E091684; Mon, 14 Oct 2024 04:00:10 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E47583F51B; Mon, 14 Oct 2024 03:59:38 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 05/57] mm: Avoid split pmd ptl if pmd level is run-time folded Date: Mon, 14 Oct 2024 11:58:12 +0100 Message-ID: <20241014105912.3207374-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20241014105912.3207374-1-ryan.roberts@arm.com> References: <20241014105514.3206191-1-ryan.roberts@arm.com> <20241014105912.3207374-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: xcuco3a8m7a9eqnj6h6hfw9wjymw9icb X-Rspamd-Queue-Id: 170DAC0009 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1728903575-642657 X-HE-Meta: U2FsdGVkX19d5QI2hyIreMKRHPC54Q8zP11rr3IkoByiDWSgYosDlJzsWAwAsMnkHzOUzdK3v5xLyYtY6GYmEkGn8FAh92DJCyhIoXq9B/B2QIyM65jYZVYx0voKKgCWxsHyVMXuhveIk1QWYxjSUFrwSgC5awNUTcROrMpzlMXBU+UvuR74+lXw4zY9h2NyNp+6bM/KuInyU6NsKAEGpxGqySyTvOsf2zSl4NOyPJTt63q3TNSZOADsSVC8wjkD3J7UVsPYz5VVNAsUOju+GGtDJYCuAbj02XW26ecFPlLaFGkPvDmXTomu802Z3OiX/Pb2Kzh4dXQKoJ3z5ExT7/00bWlLg8x3K+bogdnbJ9HsUmgUbqYHR//UD6Qd3tNVvxj6413vBrmr8ihgN+vvBMTvVhYT5v/q1bGoPLJVm4XV0K1Nn+VZfW7o1SkqyafLqBiVJL/UhvS2UCiNUjH1rsMtmNxUNjg59VNwSKbcxmQhCJe/Gf2Y3UyjFu09Y0G2cb0KeFjSMC9CUxHv9orWFFkda/X7smElOpX7XWLMKbyYvr4VmDBRk6A+QQOGxU11k1qOCRWKW9k0lG+0yQWE8tNrcgU6lK3uM76iz0CEK1Qp7fjgm3bB94u+mnZM/McKu1uih6IbPDKcy908TCInU8l74Td0bAzwZAwcX8l6Gs+cjHRi3P3/2HCRIuT1gRgqbM0ZD7GtbfDzRoz0oiQUafnvOECGBymVoRyf76Q3vy1QK3i0EyRo7Jzz2gD0V4J0iuGi/ztV+HYz2i9svGY9MnpvvhgzOj3yBGvWg9r3cErLL+voz8K4hWwdHsclKWQ/YGI3lUrjPsGXvC8L0xMxLFn/FAlB6X15gQ9QdrTozDvWPVLonR53M9HbKUJlPSlHZLN0i+dQWNZgWkmZ+q3tpf4FnYnSdXv7QghEbh48cDBDhYipmbUM5XtVR2Q8Bg5NiWgFBQ1ST/+VzyHotUs vTV/Ca6D vPm2oedOwWg5UK9tEVpB7KK6Xg/c/CjGt0dxoaqfkD6EhyZp4f2pBdOmk3HwGkrCmRQkU1wdJLTnQC5qxsz6kik97aRsYTuU6MSpL7qBst/7+egrn6ih7UWvPza0oopgXnB6w4C6It8h2AlcjoBGv29lkmQ+MbR0gkZ2gKApcP+I9xIgeryTXBbCnww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If there are only 2 levels of translation, the first level (pgd) may not be an entire page and so does not have a ptdesc backing it (this may be true on arm64 depending on the VA size and page size). Even if it is an entire page and does therefore have an entire ptdesc, pagetable_pmd_ctor() won't be called for the ptdesc (since it's a pgd not pmd table) and so the per-ptdec ptl fields won't be initialised. To date this has been fine; the arch knows at compile time if it needs to fold the pmd level and in this case does not select CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK. However, if the number of levels are not known at compile time (as is the case for boot-time page size selection), we want to be able to choose at boot whether to use split pmd ptls in the pmd's ptdesc or simply fall back to the lock in the mm_struct. So let's make that change; when CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK is selected, determine if it should be used at run-time based on mm_pmd_folded(). This sets us up for arm64 to support boot-time page size selection. Signed-off-by: Ryan Roberts --- ***NOTE*** Any confused maintainers may want to read the cover note here for context: https://lore.kernel.org/all/20241014105514.3206191-1-ryan.roberts@arm.com/ include/linux/mm.h | 15 ++++++++++++++- include/linux/mm_types.h | 2 +- kernel/fork.c | 4 ++-- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 1470736017168..09a840517c23a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3037,6 +3037,8 @@ static inline struct ptdesc *pmd_ptdesc(pmd_t *pmd) static inline spinlock_t *pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) { + if (mm_pmd_folded(mm)) + return &mm->page_table_lock; return ptlock_ptr(pmd_ptdesc(pmd)); } @@ -3056,7 +3058,18 @@ static inline void pmd_ptlock_free(struct ptdesc *ptdesc) ptlock_free(ptdesc); } -#define pmd_huge_pte(mm, pmd) (pmd_ptdesc(pmd)->pmd_huge_pte) +static inline pgtable_t *__pmd_huge_pte(struct mm_struct *mm, pmd_t *pmd) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (mm_pmd_folded(mm)) + return &mm->pmd_huge_pte; + return &pmd_ptdesc(pmd)->pmd_huge_pte; +#else + return NULL; +#endif +} + +#define pmd_huge_pte(mm, pmd) (*__pmd_huge_pte(mm, pmd)) #else diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0844ed7cfaa53..87dc6de7b7baf 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -946,7 +946,7 @@ struct mm_struct { #ifdef CONFIG_MMU_NOTIFIER struct mmu_notifier_subscriptions *notifier_subscriptions; #endif -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) pgtable_t pmd_huge_pte; /* protected by page_table_lock */ #endif #ifdef CONFIG_NUMA_BALANCING diff --git a/kernel/fork.c b/kernel/fork.c index cc760491f2012..ea472566d4fcc 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -832,7 +832,7 @@ static void check_mm(struct mm_struct *mm) pr_alert("BUG: non-zero pgtables_bytes on freeing mm: %ld\n", mm_pgtables_bytes(mm)); -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) VM_BUG_ON_MM(mm->pmd_huge_pte, mm); #endif } @@ -1276,7 +1276,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, RCU_INIT_POINTER(mm->exe_file, NULL); mmu_notifier_subscriptions_init(mm); init_tlb_flush_pending(mm); -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && !USE_SPLIT_PMD_PTLOCKS +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) mm->pmd_huge_pte = NULL; #endif mm_init_uprobes_state(mm);