From patchwork Mon Dec 4 10:54:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13478199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66A8CC4167B for ; Mon, 4 Dec 2023 10:55:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F205C6B02B8; Mon, 4 Dec 2023 05:55:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ECF566B02BA; Mon, 4 Dec 2023 05:55:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D970B6B02BB; Mon, 4 Dec 2023 05:55:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id CAC256B02B8 for ; Mon, 4 Dec 2023 05:55:53 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 9BA0E1601A5 for ; Mon, 4 Dec 2023 10:55:53 +0000 (UTC) X-FDA: 81528830586.30.5F19EFE Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id DC66D4000A for ; Mon, 4 Dec 2023 10:55:51 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701687352; a=rsa-sha256; cv=none; b=csZ6zXc2022UUhmeFY7GukymaHqH9bCJr7tvXMYHQAx6KxaSRpDgkKeMwPC6GpEMH8G1Ty Wk/Ss1kqsWvMvPrmh357PEH6OeNwnp62x5/6ZDwVfVnTMx4Dehg+yxAZdIG+yBd6e0vpM6 aMbgNetKtP2pzhM1U2fL1XYmLrxIW6k= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701687352; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+nCRyAXXqihs+BnNmeqcBD6lxH4FGaiu3Zd52hGhiqQ=; b=8bCMyQm6F3pB8H4HVUM1OdLdTSOKInIoF87q+FVfzpchXRSeT1khE9TkhDwABKIavrMODG G1LaDP/la3tRYly3L7JOJeNjK3g87v++OkjEJFgcfikF3fsMGsM6iD85HQ9RvSpwrGMsri qZn1s0PN+gCdH3LI+LbeLFZWz8YCxAY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7ECBC1650; Mon, 4 Dec 2023 02:56:38 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C9E2E3F6C4; Mon, 4 Dec 2023 02:55:47 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Andrew Morton , Anshuman Khandual , Matthew Wilcox , Yu Zhao , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 15/15] arm64/mm: Implement clear_ptes() to optimize exit() Date: Mon, 4 Dec 2023 10:54:40 +0000 Message-Id: <20231204105440.61448-16-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20231204105440.61448-1-ryan.roberts@arm.com> References: <20231204105440.61448-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: DC66D4000A X-Stat-Signature: 8jkt9aw8rasmpw3ajjdb8s3bf9extxbp X-Rspam-User: X-HE-Tag: 1701687351-845665 X-HE-Meta: U2FsdGVkX18WGzbY5UWUi1Qh7GJDhOqrHbAw/hoGj/cMitUfxPcqUBO0tspWfwIQ4lecenzCskDEqFipgucolYwOrSlhDbbxoRMwM9vkiAV9QP3rOPpEWBXl92PO5GDqVMLI/cW3CmvODaESFWnlq5EJMKCr5AHjrsz6EWzlwlhi470163We0ioqJzjwlNJ93qJK9jPOku88hCx5JbxGVC9RbXS3woK6+lubXGyeBPVkN7TsxyYMqojPxzoMjaDmQSCTFBp4ClCnFSEgMHLSk3yZhxPK9d9N3RK4VPQ5RnUk3UAL077UEVZZ+Hl4nx3sgDn0ao2ZdwlUBJgIPUWa/6Yp3rxUbOLL7CHxMcP/RXrG7GcAXX49U4aSrZUjCaY78EaEvo0/jDqDhvW15Gtem4UW/FWSBddU3LZc8ZqhAyR7WKeFqpXQC91VRwiXS/62Ug0WTF7QvfUY1rUJPj1DmxYVkVNPwQTXR/EWJNOnU8yaYOYVj2lT1KqraKT8Y9MOS/tBl7r0qxBIK8f7o/1EkTrs+H3TQYGZGBH9AZeyv0PxCgAwvgl3/S2Iy3RHD0pIIF6OaH3GLnV6mip4kaKZg1gdb3S+Db7E/noUCAzCeymTvE+DHxoBPV/sNSMyVgweb3gt8Ew3n3ZsK+jHsjvr7ZfCD9gisvfRx6V+EVnwU9Bt6Hd1jN07vXPJ35Q9Y0PwsZv8BLLSVqczwFzGxnEntHZhWXW36CaejSqFUJO73uPX1RXGi17WfVRibl0bLMUTLij3iXq2iQAye15vdgGoSBNG0/KRo5rx8YDQSRKnIDX14mntehOE23gdb9o4JicbdeYM0balgK2G3+yJHlmI6jW2hCt6o5wR5quM360vP+gLKzoi38U+RIdeeD/CXvAMeOowrBog70PlEx+lpBXmlYan5saxu1gtOGvDFRUhpfSeTU7YVcaMSjIwUIx74IYcOulpMQuuNJY5I44/jM6 nNOC5j+h NgteGk4+lO6sABaYC2WsrYoLdEiPsF6hNEs4u4lnMPCCDAbi/v3Bb+7F4+CqZzlyk9vzURxIIcF1plt4CrCAGK4PG7McB2CricTL1HirotQo2d2o0rMglgMkd1OQvS6aEH3URu9tjiVKevsQ2mToD1UcnQ7WhBJxXWt+e2lv2PAqwvZ1kGUTRAzl3PyhnISPrP9ejHXYOI/o9ezlQ3ncNAmgVUQybFoONJ3xpgf/Q/x8ABtX8ubYuTk2IL585VWC500LirYyQtMUI065igkjLb0QvTg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With the core-mm changes in place to batch-clear ptes during zap_pte_range(), we can take advantage of this in arm64 to greatly reduce the number of tlbis we have to issue, and recover the lost exit performance incured when adding support for transparent contiguous ptes. If we are clearing a whole contpte range, we can elide first unfolding that range and save the tlbis. We just clear the whole range. The following shows the results of running a kernel compilation workload and measuring the cost of arm64_sys_exit_group() (which at ~1.5% is a very small part of the overall workload). Benchmarks were run on Ampere Altra in 2 configs; single numa node and 2 numa nodes (tlbis are more expensive in 2 node config). - baseline: v6.7-rc1 + anonfolio-v7 - no-opt: contpte series without any attempt to optimize exit() - simple-ptep_get_clear_full: simple optimization to exploit full=1. ptep_get_clear_full() does not fully conform to its intended semantic - robust-ptep_get_clear_full: similar to previous but ptep_get_clear_full() fully conforms to its intended semantic - clear_ptes: optimization implemented by this patch | config | numa=1 | numa=2 | |----------------------------|--------|--------| | baseline | 0% | 0% | | no-opt | 190% | 768% | | simple-ptep_get_clear_full | 8% | 29% | | robust-ptep_get_clear_full | 21% | 19% | | clear_ptes | 13% | 9% | In all cases, the cost of arm64_sys_exit_group() increases; this is anticipated because there is more work to do to tear down the page tables. But clear_ptes() gives the smallest increase overall. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 32 ++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 42 ++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9bd2f57a9e11..ff6b3cc9e819 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1145,6 +1145,8 @@ extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); +extern pte_t contpte_clear_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, @@ -1270,6 +1272,36 @@ static inline void pte_clear(struct mm_struct *mm, __pte_clear(mm, addr, ptep); } +#define clear_ptes clear_ptes +static inline pte_t clear_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, int full, + unsigned int nr) +{ + pte_t pte; + + if (!contpte_is_enabled(mm)) { + unsigned int i; + pte_t tail; + + pte = __ptep_get_and_clear(mm, addr, ptep); + for (i = 1; i < nr; i++) { + addr += PAGE_SIZE; + ptep++; + tail = __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tail)) + pte = pte_mkdirty(pte); + if (pte_young(tail)) + pte = pte_mkyoung(pte); + } + } else if (nr == 1) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + pte = __ptep_get_and_clear(mm, addr, ptep); + } else + pte = contpte_clear_ptes(mm, addr, ptep, nr); + + return pte; +} + #define __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 2a57df16bf58..34b43bde3fcd 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -257,6 +257,48 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL(contpte_set_ptes); +pte_t contpte_clear_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, + unsigned int nr) +{ + /* + * If we cover a partial contpte block at the beginning or end of the + * batch, unfold if currently folded. This makes it safe to clear some + * of the entries while keeping others. contpte blocks in the middle of + * the range, which are fully covered don't need to be unfolded because + * we will clear the full block. + */ + + unsigned int i; + pte_t pte; + pte_t tail; + + if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + if (ptep + nr != contpte_align_down(ptep + nr)) + contpte_try_unfold(mm, addr + PAGE_SIZE * (nr - 1), + ptep + nr - 1, + __ptep_get(ptep + nr - 1)); + + pte = __ptep_get_and_clear(mm, addr, ptep); + + for (i = 1; i < nr; i++) { + addr += PAGE_SIZE; + ptep++; + + tail = __ptep_get_and_clear(mm, addr, ptep); + + if (pte_dirty(tail)) + pte = pte_mkdirty(pte); + + if (pte_young(tail)) + pte = pte_mkyoung(pte); + } + + return pte; +} +EXPORT_SYMBOL(contpte_clear_ptes); + int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) {