From patchwork Thu Feb 15 10:32:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13557864 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9150FC4829E for ; Thu, 15 Feb 2024 10:33:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96EDC8D0020; Thu, 15 Feb 2024 05:33:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FA1D8D000E; Thu, 15 Feb 2024 05:33:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D4AF8D0020; Thu, 15 Feb 2024 05:33:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5747D8D000E for ; Thu, 15 Feb 2024 05:33:10 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 288B4A2690 for ; Thu, 15 Feb 2024 10:33:10 +0000 (UTC) X-FDA: 81793675740.25.E68C4E3 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf11.hostedemail.com (Postfix) with ESMTP id 81CB540002 for ; Thu, 15 Feb 2024 10:33:08 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707993188; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BCAtce4Ja9jmZhUdX/dylO55pAFHXc3eVtMKGJQHUkc=; b=TBiirKtuhw1fnB25EZN8ZmFkLawqnLs51bDFsA5lUpmv3Q6m4g1U7VZgvL09CXpNGfwbgY xJb4vS0Almt7oiUGcq6R/fBAL58tchXYLi73csSn1qfQKeoT3tLYDUoM6gpmR9z8VNpgaf l2OQC4ccn0BwcoDbtf93nepsotVPmNU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707993188; a=rsa-sha256; cv=none; b=EyA9AXWBFokpekNQRV7/qTowlmmgNapO/YXguZtGczfFZeZfcJx7gL1KNknC9Ibjt6pFgD 0fapHO2MGVsZ0aK0gqd4sScxQm7fiqG2eTjTHdYKbnJBNmZj+7GAD/QdyyUQMdKIZFpf7G TUS3SJA3sUjY2O5lH5Zt1vyCnTXC7A8= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B0CA71596; Thu, 15 Feb 2024 02:33:48 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A85623F7B4; Thu, 15 Feb 2024 02:33:04 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 14/18] arm64/mm: Implement new [get_and_]clear_full_ptes() batch APIs Date: Thu, 15 Feb 2024 10:32:01 +0000 Message-Id: <20240215103205.2607016-15-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215103205.2607016-1-ryan.roberts@arm.com> References: <20240215103205.2607016-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 81CB540002 X-Rspam-User: X-Stat-Signature: bbs3uhzji8eenebznt8fad7cycscroc7 X-Rspamd-Server: rspam03 X-HE-Tag: 1707993188-147889 X-HE-Meta: U2FsdGVkX19oS5AL52uWx7vmSHMWqCN1yUBwFwywpKCMw4TOHrDwrzB4hsxvh9AjlukQrN8qGYrdVdcoG2LTWWixSOlQaLemDgKakrjc3a2tgoSIpYsZJTrtVgnj8LRQKkpu8SS5/bKji5H8WWa38qTt7P5IRemgozgQU4Ytcp+jnHpBrQdDkeiRmj/NCtftiaDalxOTOmICD8TBiHysGldI7FNwpxkpH0zAvIRniC4N8mqcxIA+u9fwUWdIiNPUh2EcZ5ATg1xiguaS0uNndR0H0HRBtCSEFHP/3y5QBZWrW+hDLwM2iwyvEov8NdF9Y3b+o2tXk89Wlul95IR1SsT4/U55Zni0+1TtYtlOtEfvXhkgAt6Eo3YpE8HWar8sfCaCf1W9nprMooTbzmlVybGVgCAa5D5IXVoldjAXM3gDqmIGhbp7S2xQWDmsEdO9b+c4c0blOOoQMSq6nqWGAlorse3wvTo5ROk5nq4vu2sua7CUsY0W3yO50Bvm/S29z8HPh2yWPDV2/ro4M+4pYy9bQogxtVAxZoWobqD5IXjaP2vGa14icCK3lJDgC//zFtAAUx7+BSi1c68dH90LHxgcJRkTCfk6hsv+3/3IbzL7+uciO9KeawLGplLps9Esl2gL8s2cV7ZlbDQjCBYcKOy3PjE6m4Rhk00/Zjcu1aYfE45ILvW86tKolHpBUizkmSujtLVs3Fc5WZ9uYTW2wBm7EDAYN351JfbxBEr6m4Bg4sSCBfrH225Ep2zFuOaqpbFiWmRHOwaR58UEQ5c+x5F0GnqR57jQjQYFjoZ70M0qQPwQLDqtvDlITBmxJy8ScH75B9QiVahPIe2RjE8HC0IFFXwOX2DWcoCW2e3x67DkwGiYozTjkA644vuDthbJdJb6rXt+vK27rCA9LQ3jJN/GLUbhrQnHLYNIM7XieAnOumJkOpUPxWkQhUCNbCwWnLI2E+CE4kJDCkQRMpI jyXkPZbK 5JmSGwqMdiRGp1G+kRW06sy7ct39r7vz5Zt6kc3LkRlDV0WUYgPGwGYoPyYjWae37p9PGT01otVwT6xrWHvKwe5t84RdluHSffudaNS9KPF7aVMVf2ZjvI0ikw/uufubWiwf3/Mo0dN2LOPzSXixT+k9zmn60Ndd12FKTBB1d3jqZYa7J5SG/IarxtvSWqrx6vxMfrApXj5QOiGEUfPRgOIkQmQUJ8eDKD1gEMFKFhXEzFCyBPDrABxXHpbppi/Hq2CE17sTE/2BV2TBaih4GpjCXTnjV1PqMQ+Urfs2CcN/oejc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Optimize the contpte implementation to fix some of the exit/munmap/dontneed performance regression introduced by the initial contpte commit. Subsequent patches will solve it entirely. During exit(), munmap() or madvise(MADV_DONTNEED), mappings must be cleared. Previously this was done 1 PTE at a time. But the core-mm supports batched clear via the new [get_and_]clear_full_ptes() APIs. So let's implement those APIs and for fully covered contpte mappings, we no longer need to unfold the contpte. This significantly reduces unfolding operations, reducing the number of tlbis that must be issued. Tested-by: John Hubbard Signed-off-by: Ryan Roberts Acked-by: Mark Rutland Acked-by: Catalin Marinas --- arch/arm64/include/asm/pgtable.h | 67 ++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 17 ++++++++ 2 files changed, 84 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8643227c318b..a8f1a35e3086 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -965,6 +965,37 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, return pte; } +static inline void __clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + for (;;) { + __ptep_get_and_clear(mm, addr, ptep); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} + +static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte = __ptep_get_and_clear(mm, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, @@ -1160,6 +1191,11 @@ extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); +extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); +extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, @@ -1253,6 +1289,35 @@ static inline void pte_clear(struct mm_struct *mm, __pte_clear(mm, addr, ptep); } +#define clear_full_ptes clear_full_ptes +static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __clear_full_ptes(mm, addr, ptep, nr, full); + } else { + contpte_clear_full_ptes(mm, addr, ptep, nr, full); + } +} + +#define get_and_clear_full_ptes get_and_clear_full_ptes +static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte; + + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + pte = __get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } else { + pte = contpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } + + return pte; +} + #define __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -1337,6 +1402,8 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define set_pte __set_pte #define set_ptes __set_ptes #define pte_clear __pte_clear +#define clear_full_ptes __clear_full_ptes +#define get_and_clear_full_ptes __get_and_clear_full_ptes #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index bedb58524535..50e0173dc5ee 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -212,6 +212,23 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL(contpte_set_ptes); +void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + __clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(contpte_clear_full_ptes); + +pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(contpte_get_and_clear_full_ptes); + int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) {