From patchwork Wed May 8 19:19:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4E43C04FFE for ; Wed, 8 May 2024 19:20:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 612F86B0093; Wed, 8 May 2024 15:20:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5C32E6B0096; Wed, 8 May 2024 15:20:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4643C6B0098; Wed, 8 May 2024 15:20:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 24CF36B0093 for ; Wed, 8 May 2024 15:20:39 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5179CC0CD0 for ; Wed, 8 May 2024 19:20:38 +0000 (UTC) X-FDA: 82096195356.24.BD4D1F6 Received: from mail-ej1-f43.google.com (mail-ej1-f43.google.com [209.85.218.43]) by imf03.hostedemail.com (Postfix) with ESMTP id 769A820008 for ; Wed, 8 May 2024 19:20:36 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=SqRejkkG; spf=pass (imf03.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196036; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=datyv3e5LLeWOM64wvZ1mfI1WId3P2rVazcirFn2Oxk=; b=Mq5OPfzcHiQvpeBmbMr5IPbF/qsvlSFNEmDvs58ZA8WtLEnyh4KfxrOeaGOfzBnYWD1287 +rNBQK/wRDZqF8FrLX6OpJdWaUFD3qGC7dtMOAtfZtZaVJUia2hVSbDES7xbAYk2DWDGPm zFLZWdFbR8RjZQKEIxy62OFvErSc46A= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=SqRejkkG; spf=pass (imf03.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.218.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196036; a=rsa-sha256; cv=none; b=NqN4X1GPN/RC0YWbx0/iNpennWAFtp8p/m/TpuU9F/XoWdSN8JozgOQbpXL67XAyAooibJ RFXUd64WNvdsrIdk6OI7xLDFcXc4+5VcolOiLfaobtwVEBTjPF0gxTMveBkhBv56wtNgA5 cxPlFSovpVNVBKn5eOQpF9FYS6/8iPc= Received: by mail-ej1-f43.google.com with SMTP id a640c23a62f3a-a59cf8140d0so20571166b.3 for ; Wed, 08 May 2024 12:20:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196035; x=1715800835; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=datyv3e5LLeWOM64wvZ1mfI1WId3P2rVazcirFn2Oxk=; b=SqRejkkGM8sx1s0h0nncPnhR7M0bkDhVNWMzdf7Dte/botXJ8Hhtd+C33m5c6Z0ppS gq77Ag99mhFWeyOWjdQdSsEZcwP2hH63rs/oOJBtJSXrXT+4uTfJ7fU4Plj7wvJfTFu/ +0+CLejNvHj4amE2P6Pvj/eLJx2YmMjGny7rEU8r5txqjpC8S0p0eHuiW32hrXUPzbJt v76ZslVLcq2S7GceJIdLq25hUp0DoL0GdslWPm6f4f4fRNVDhH4Dr6kDxFkr0G5CNfsv vUhD0BkLjGhAV5QHZFann4AgLtFw8Air5gbnEw2P3eJdZiFqujASCdZhVqWKsbjLKmms ML2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196035; x=1715800835; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=datyv3e5LLeWOM64wvZ1mfI1WId3P2rVazcirFn2Oxk=; b=ozwD6kMQSUNO6aNSTH6fhxuXU9H9T6LGg2vzkBJXFUlQCU4RIpw4l5huGsTOGRmCKi 1TL2MDmSy4qNN79fRj9z8pY5+116JJL26AaTTfPF864kDJJR3i36vJUazke9JjL79jz0 OeS7npd34RMiwVKvl0WKhk3NgEfhsyJuPdW0yTwT+cd1ZsYAkYOQcMiJ/NPySZ3pjS5b +Ee3z9u8K4RZdNFYI/vpzGu/rHpy0ShmgiC7gxKlb7GCxGcrDKpaV5ruq8y7DiXeyn3z y00dufDV1IyMMS/wwCQA6UOSd8uOFOycQ7tVqew4DFBGlEjiRLm1YFKmzJXZXdelRG97 y8bw== X-Forwarded-Encrypted: i=1; AJvYcCXHwP+cpuCOC4C0j2nSiERE4gDddCbmu0OXNFIWcpffOgIe5CEyLYtCOpoDMu74D/pDp87p3g5xFsEcV6zgFxuRwFg= X-Gm-Message-State: AOJu0YyrlnALEOW5X/FUB02ZNdP14IQHH+qBOipeGT/lrOsK2sF20e+Z hCjb+1lm6Pvc5w7dC02ZzpUtdhu/Gx0TWZUa+5ds6nro70U6x042V778Ma05isg= X-Google-Smtp-Source: AGHT+IFPtUz+3X9HHx3rOZWYiSnLgxFfxDjotG1sGB7J9SeveUOh2JXAABXt+ni1vfq6RI0lDXdHfA== X-Received: by 2002:a50:ab59:0:b0:570:3b8:a990 with SMTP id 4fb4d7f45d1cf-5731da6977emr2508293a12.39.1715196034646; Wed, 08 May 2024 12:20:34 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id g17-20020a056402091100b00571bbaa1c45sm7881992edz.1.2024.05.08.12.20.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:20:34 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 01/12] mm, arm64: Rename ARM64_CONTPTE to THP_CONTPTE Date: Wed, 8 May 2024 21:19:20 +0200 Message-Id: <20240508191931.46060-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 769A820008 X-Rspamd-Server: rspam06 X-Stat-Signature: 7buu748ykp3fgq1kejkayfuwmz6bedey X-HE-Tag: 1715196036-316996 X-HE-Meta: U2FsdGVkX1+KADS95n2JDFw/9HarMfTBlYZ1J5+fvHBuGHtELgTy/8oXCVtoHCafctNscAN5cNMtQcFPXpPmL7fJpZgkOCKVkrOeimgr+khNVoARYHs541S/DMTBp0z4fZXGnkAbvlaNbFQSL9jtwJ9A8gktfL1vtu7z7nJRjJvDK8exTCHlj59KVc6MUB0gqsA+9oT0UUg+/B69ig6Z2MR9VK4Q5AisNrsK6+/wI5jP8P5W3GpYC6BlqEZnxxYAxRfXYanWLgv0gqmR8ILyX1zVx4+ANax/FGqvMRggczfUlRdZOBmicmhgcqS4Qgszl0p41SCv09rX9jrmOz0sfqVm0d31YrpyYcUjIjGto4kJTYh6fuEwJoV+kxOnwTay3Eh/m4gwptoj8Ucy9ufOur3rugg8rddvqSvlcV9sXM9rZmv+DaPM9SO9OLgJoOnqXTKNC7VMan1PEgkRkawvc1k54o+gDA11hovPkSoNZA89CLwzUYqf5hBvpo0TPekoN04lYGMbdacYvyUKk1mydDla7IC/F79ts8FhIab66c16GmwFW0k0vluTpeAeSNDq5RiDpAwliUqEXuPwcrWyqfPIZ+bodnWbAVo8iM28ux9fzRezVy6s9jKGQBRo9GH2Ndfn62IIW97towJEM0AafAI2dsBc8rI9v0ZvAz/XaNH0J8JWFuXjA+lgB2AQ5G7d3SF6e3FTrV/t+jcSKU5iAmpjFW9hHC2o3dH/SILqCvg6nDeuf/8YU3NuW4swkt7nH2W8SHBWrUyY7/JqM4iCjpEmxrSh5xtSgQcjkVB0TCRJpOrCmO2q0dVRzUpQolrx7oD7pWIG0yo/baHAWR8wuKXX+zDt5r7GDb2rKK3CV5nP5c4y6G4dAr2BVS2ANSEMVhmW8JGO3PNSM8ekpmbMkoIcTLk60QaIrptJ/dhyvfEBGMKaE+lvLftStuYsq2NRPcInqBxgCGxn4tg9ASV gNPF3e3D KtPkCG4FMD+ijRYlSZ6n9jUBM/emGoLKH72adwOjSE7mOG80/DhID0+eTRe9Om4ojabWJbIrtMXdGN9lbabQ+dDaCoixTtrgpxD73hbgPmq2XRfHjUdiclLhFsVxYqp28ojYuoLiy+uPaKrZFVpNVz+WKxJaDaM/BYc0L1SfazgtG4z2NbY+h/zjbLCx5dIkksRSLOqKqCg29tu6jcoMuDcD3Sx7U520Aj+2mX2N21ZSRkQZ1Sf8mvW43xWGwbRIY3m1H7pj7ixKQ9Z/lm9kmTxxMUVO3tLO2nHYgiNrezUrt9WNcb04/sM5+iCpB/HB4ntRAWzPra4GJTOCSv5i3hyoTKJ9bsK2lrDC5KuBjeldVGbEcBefSr2uxkpuVi/mJBXNG4n2eiqylU09L+L27b7n75oBS/EEEAvGk8FzjlLus88R6bb2IEe3skDgZHFFS69ElR9VcGF+pi0EBsmNwQ0rhDK7/V93QhovDY+EDCBpBOuUgZLNVFkfvKqrIateTVPeMRgp0Z6QpTzXI1k0aYmZiH4EWl4VOS/IdAkNRsPSqS/+hC+Fx/x1mFw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The ARM64_CONTPTE config represents the capability to transparently use contpte mappings for THP userspace mappings, which will be implemented in the next commits for riscv, so make this config more generic and move it to mm. Signed-off-by: Alexandre Ghiti --- arch/arm64/Kconfig | 9 --------- arch/arm64/include/asm/pgtable.h | 6 +++--- arch/arm64/mm/Makefile | 2 +- mm/Kconfig | 9 +++++++++ 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index ac2f6d906cc3..9d823015b4e5 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2227,15 +2227,6 @@ config UNWIND_PATCH_PAC_INTO_SCS select UNWIND_TABLES select DYNAMIC_SCS -config ARM64_CONTPTE - bool "Contiguous PTE mappings for user memory" if EXPERT - depends on TRANSPARENT_HUGEPAGE - default y - help - When enabled, user mappings are configured using the PTE contiguous - bit, for any mappings that meet the size and alignment requirements. - This reduces TLB pressure and improves performance. - endmenu # "Kernel Features" menu "Boot options" diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7c2938cb70b9..1758ce71fae9 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1369,7 +1369,7 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); -#ifdef CONFIG_ARM64_CONTPTE +#ifdef CONFIG_THP_CONTPTE /* * The contpte APIs are used to transparently manage the contiguous bit in ptes @@ -1622,7 +1622,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } -#else /* CONFIG_ARM64_CONTPTE */ +#else /* CONFIG_THP_CONTPTE */ #define ptep_get __ptep_get #define set_pte __set_pte @@ -1642,7 +1642,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags -#endif /* CONFIG_ARM64_CONTPTE */ +#endif /* CONFIG_THP_CONTPTE */ int find_num_contig(struct mm_struct *mm, unsigned long addr, pte_t *ptep, size_t *pgsize); diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 60454256945b..52a1b2082627 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,7 +3,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o fixmap.o -obj-$(CONFIG_ARM64_CONTPTE) += contpte.o +obj-$(CONFIG_THP_CONTPTE) += contpte.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/mm/Kconfig b/mm/Kconfig index c325003d6552..fd4de221a1c6 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -984,6 +984,15 @@ config ARCH_HAS_CACHE_LINE_SIZE config ARCH_HAS_CONTPTE bool +config THP_CONTPTE + bool "Contiguous PTE mappings for user memory" if EXPERT + depends on ARCH_HAS_CONTPTE && TRANSPARENT_HUGEPAGE + default y + help + When enabled, user mappings are configured using the PTE contiguous + bit, for any mappings that meet the size and alignment requirements. + This reduces TLB pressure and improves performance. + config ARCH_HAS_CURRENT_STACK_POINTER bool help From patchwork Wed May 8 19:19:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51383C04FFE for ; Wed, 8 May 2024 19:21:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA3836B008A; Wed, 8 May 2024 15:21:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C53676B0092; Wed, 8 May 2024 15:21:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ACD9C6B0098; Wed, 8 May 2024 15:21:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 8CF876B008A for ; Wed, 8 May 2024 15:21:41 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 01A871A0D5A for ; Wed, 8 May 2024 19:21:40 +0000 (UTC) X-FDA: 82096198002.08.F9708DE Received: from mail-lf1-f51.google.com (mail-lf1-f51.google.com [209.85.167.51]) by imf09.hostedemail.com (Postfix) with ESMTP id EB2B6140013 for ; Wed, 8 May 2024 19:21:38 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=BlLTJBb3; dmarc=none; spf=pass (imf09.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196099; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ex1T8OVqbNmmxq9WLkEl/7It9+lEf4Ajscw14zap9bQ=; b=Vk7p7/SY5hSJVLRSL9uRTLEhXq79yb7+sqeAlL89u3+8N2OrE177xkA7fV6s0K71+PJC4h t27uwx8pLRJOnF0nkHHyZGkfsr4Jm1uW2HLJ4+t/SRyIBQkFtcOqg86iMCaMUKjT2iVqab +0jd6WS1syConiuF9r8pti8g2C4J18s= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=BlLTJBb3; dmarc=none; spf=pass (imf09.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.167.51 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196099; a=rsa-sha256; cv=none; b=Frc7fuJUMWs25XFPW/YZauK+0fELv63UO4CLNFoUyC9AkqrQXYMWB8WY8yI+4xJ0haESaz AIHN2knucmORFPCHlFjA92D4sHOLMNhQdGFe8Wi6t1uVo56xuRrwRaG9DikPsCyvuh7p9c q7oY0N0Q2mG4gmyqR7B58OsU49kamI0= Received: by mail-lf1-f51.google.com with SMTP id 2adb3069b0e04-51f99f9e0faso46285e87.2 for ; Wed, 08 May 2024 12:21:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196097; x=1715800897; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ex1T8OVqbNmmxq9WLkEl/7It9+lEf4Ajscw14zap9bQ=; b=BlLTJBb3wnkMDS6/E6eU02I5f1jqnj5dbgiJGlkltayCZg1FR2XlO8WUeHDkPHVsuX ln4biPASOo21D291CpM4IZe6yZk45nygbWDfacvv+9jUOoeAVGo32BzwNMDiKYTcopQA RjvlITH7iSdMT146rk/OmyGElGU0p0sImln55tLjQGOFKSingS3WdnG8ILStKlyuLnoA 7/nxcZDyNw8N3459yzg/eXOLDg8Sig8t+Vunr90dEgqmX7a9xOkZRKnhAd7N+sbJB2ga JQ7IXISvZYKVy+MSW2s6orFovn22y5pkmDBcnO7/fWadnP654CS0a3GmivzuP3TlrJ5B dnwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196097; x=1715800897; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ex1T8OVqbNmmxq9WLkEl/7It9+lEf4Ajscw14zap9bQ=; b=j0+Y0vfAhz4osalo2CJS+8+ciNQ4kN9couKNr1NjcEj87oUTBX8yS8ccCoNeLxVI6i GaFAxkg67RL1vwJAStgJWcUIcHWacwWubI5BVZ5k46tKVtaUtyshHVr7KE7TQxqi7I9n 0fY20Tuw1t+IUqjXxXKAessbbIJuDv0Lry+6KnwzXQtgcZKLgL7Rr3VTK+vo9L0svToc EmM7afNczIguQr873VS2976iKoEcDPxLoMSWrNHFdcde3OzvNDrhFbkNIN9Jc/RZQf2v BdbzxCO55hqct6tbn193c6mcDJ+LeyfNzhIlVulQRPDvUi91YM3eKzAVMnK4tlkux0qt TQdQ== X-Forwarded-Encrypted: i=1; AJvYcCUDW49bWYr8uURhwcjTX1zAt9Mv9yuqJxixrfQOTi6pgEZPKi4Cr8zL+lBwDlD5uWjQKNBW2hYHCUu5uv0HUF87xxs= X-Gm-Message-State: AOJu0YzZayT2/0DNOFAloTfXyZlU9oYPUdS7AtSvrqeqjmVt6hl1doGT SV3OyGAjYqwudyawDzEw1tFsJCWDp7eZ1uo56/1HdsqebQ/RosnLnuIbaRSfM18= X-Google-Smtp-Source: AGHT+IFjgJRFJyj/slQR8FaD0MqHewWXYnSXcVKmWwkcfJ76gy4SFUv7AbGxIQbLfo1flUN0YcCWiw== X-Received: by 2002:ac2:454b:0:b0:51c:8b45:c9fb with SMTP id 2adb3069b0e04-5217cd4b3e1mr2066413e87.69.1715196097125; Wed, 08 May 2024 12:21:37 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id t18-20020a195f12000000b0051f95499c00sm2324036lfb.103.2024.05.08.12.21.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:21:36 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 02/12] mm, riscv, arm64: Use common ptep_get() function Date: Wed, 8 May 2024 21:19:21 +0200 Message-Id: <20240508191931.46060-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EB2B6140013 X-Stat-Signature: rbbn9gc3f6dx7qxmkcsqgjcbweohebgs X-Rspam-User: X-HE-Tag: 1715196098-111946 X-HE-Meta: U2FsdGVkX1+Vlw2j+aSbC63hIFpXn7aH9a5HWJsEdx9Rx0Vr1WXMXtn/KWW8c501IckEGr84bW6VEfjF/EcYuVhv+oEXjpTtyCL8CHWDkYBe1Nc9jLa3GeXTJTPn2kVxX21Ijbh2v8qEoOF1toYN3S62SDjOuk0DnKbsIu0sbVQBXmdSWpG9Bnr1+Q9rBO4GBp+wCIZRmb0JbDd1EAN+wQnOiQlgMbj9xA/oiOwEqAigwk3W01PG/PoO5caDXXbSa96w+xrxcseyeuloWhvfgS/12yZH/0HWpRxln37azlPBoc009Z93zuEIvzHVVxbq+cnsATE+6OkU2oN7xF/6ozl4NZTS0g1b+3EF4kMLC344yWP1fv3Qz1TaCejdhn8NIqRZFZYWaII+WKl3MxxFy6rFduOTjffK6CWXcCyJmJNlJsg4e03CPhpwFuyZKhsl6FiyAzWM/pSfeIp7xVqijEcO4lV8VZ6eQS7IbXLdazNtOGNTUSaVzJHZ7RZGBvtYE+dYfl4qLD9OgOW99BnmXm763kMQVB5AqjsJDmDm2YkefzU88FiT4Q+NTDGbkBsOlPlOUnbkhYmJshpe9gLwLy6Usa2Msj/peuhXWeyo+1wseHR9UBUpwgoSJqP5i9cykBLJyrr4zZR6D2hdTE7vi5g2/BaWptQ5rnF4SfBBxip4gUsvceMCFWOcbLoDkcZ4v7eLrUCmBHqperPPmF05L+v5EN91PxG8Sk5xA2okKgeu57f4+9vhOYauU/AllU8jaexcfYza93zy8Lg5YH1vqzmO/CEf4ydHg5m9MF/gFxddwg2DqU7mFMMVzo9Gc5AXtiO9b/bTSQkyxdn8L/ZK3oovcV1X4FKtvh9i6YWxo9rS/uUoJpawwSk6RHvOzcEWonj+CLdnKU5/3hd5OHwJC628UOiKN1hOVF14bBXcmBSPMoJiDKpBPsnEt4yD7MZ72fTaoeAbouyE1FaFd8W Gnnj/MPe Z52lEE1A7IM6dhiNex5U0dHpDPJbhQIVPwwpljMcqSYTTSYHzY2Zel9+Cee64GGoMIk10rfDDGAcxGQA5T7gq5QXcpKt/qZi6h3GpyDFzlpmVMETgZS+yMKL+wGre/+W2A9rznoKRuR6gGRzstGYYQc22o5y2n2wyiFKdZyOPG0y8hxQpUNk6PVg0viGPA7VW0RrxR5IpNxLRVX6rjd6kcEKTW3Fa/ehjwe+LV/7MmDvahyz4MoprK3kT2CIB+o6QLxKp+yrIzZmd3Ii+OXlb1YDtdJdwni0ADcPpqUYziLcIDDyDM1TxvCPbr3TTxwY7n7/CjqEpJ+MhvjnEjFlxjkf49FRKwO5uKpjmV/tea1W6/m01ac35qS5TfsYIl7oGxGUb+1YjzWJOj8xQEputeEG525j7AsuZuqTk3WGosDD9FZ6viHRGOxHCcjQDyxNTmBwBcoP0QY5bZf/bBuTkvc/+rDbjaf8P2t9dYY/xFjIxR0xq3dUOy4WE0/OzvcZmhhxWl9W7gxDKFig34m8B8D0pph1fWyBdpZ3+8NTQaNvS05+xVzj5L6bj8g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_get() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 30 ++++++++++---------- arch/arm64/mm/contpte.c | 47 +++++--------------------------- arch/arm64/mm/hugetlbpage.c | 6 ++-- arch/riscv/include/asm/kfence.h | 4 +-- arch/riscv/include/asm/pgtable.h | 22 +++++++++++++++ arch/riscv/kernel/efi.c | 2 +- arch/riscv/kvm/mmu.c | 16 +++++------ arch/riscv/mm/fault.c | 2 +- arch/riscv/mm/kasan_init.c | 2 +- arch/riscv/mm/pageattr.c | 4 +-- arch/riscv/mm/pgtable.c | 4 +-- include/linux/contpte.h | 12 ++++++++ mm/contpte.c | 45 ++++++++++++++++++++++++++++++ 13 files changed, 122 insertions(+), 74 deletions(-) create mode 100644 include/linux/contpte.h diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1758ce71fae9..a878735deb9f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -38,6 +38,7 @@ #include #include #include +#include #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE @@ -1379,8 +1380,7 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, extern void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte); -extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); + pte_t *ptep, pte_t pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); @@ -1456,16 +1456,8 @@ static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) * setting it in the pgtable. */ +extern pte_t ptep_get(pte_t *ptep); #define ptep_get ptep_get -static inline pte_t ptep_get(pte_t *ptep) -{ - pte_t pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(pte))) - return pte; - - return contpte_ptep_get(ptep, pte); -} #define ptep_get_lockless ptep_get_lockless static inline pte_t ptep_get_lockless(pte_t *ptep) @@ -1659,9 +1651,10 @@ static inline int arch_contpte_get_num_contig(struct mm_struct *mm, * find out the number of contiguous ptes. */ if (size == 0) - return find_num_contig(mm, addr, ptep, pgsize); + return mm ? find_num_contig(mm, addr, ptep, pgsize) : CONT_PTES; - *pgsize = size; + if (pgsize) + *pgsize = size; switch (size) { #ifndef __PAGETABLE_PMD_FOLDED @@ -1674,11 +1667,13 @@ static inline int arch_contpte_get_num_contig(struct mm_struct *mm, contig_ptes = 1; break; case CONT_PMD_SIZE: - *pgsize = PMD_SIZE; + if (pgsize) + *pgsize = PMD_SIZE; contig_ptes = CONT_PMDS; break; case CONT_PTE_SIZE: - *pgsize = PAGE_SIZE; + if (pgsize) + *pgsize = PAGE_SIZE; contig_ptes = CONT_PTES; break; } @@ -1686,6 +1681,11 @@ static inline int arch_contpte_get_num_contig(struct mm_struct *mm, return contig_ptes; } +static inline pte_t *arch_contpte_align_down(pte_t *ptep) +{ + return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 1b64b4c3f8bf..d5512ebb26e9 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -21,11 +21,6 @@ static inline bool mm_is_user(struct mm_struct *mm) return mm != &init_mm; } -static inline pte_t *contpte_align_down(pte_t *ptep) -{ - return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); -} - static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) { @@ -34,10 +29,10 @@ static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, * of the range. */ - if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) + if (ptep != arch_contpte_align_down(ptep) || nr < CONT_PTES) contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - if (ptep + nr != contpte_align_down(ptep + nr)) { + if (ptep + nr != arch_contpte_align_down(ptep + nr)) { unsigned long last_addr = addr + PAGE_SIZE * (nr - 1); pte_t *last_ptep = ptep + nr - 1; @@ -54,7 +49,7 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte_t *start_ptep; int i; - start_ptep = ptep = contpte_align_down(ptep); + start_ptep = ptep = arch_contpte_align_down(ptep); start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte)); @@ -122,7 +117,7 @@ void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); expected_pte = pfn_pte(pfn, prot); orig_ptep = ptep; - ptep = contpte_align_down(ptep); + ptep = arch_contpte_align_down(ptep); for (i = 0; i < CONT_PTES; i++) { subpte = pte_mkold(pte_mkclean(__ptep_get(ptep))); @@ -152,34 +147,6 @@ void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(__contpte_try_unfold); -pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) -{ - /* - * Gather access/dirty bits, which may be populated in any of the ptes - * of the contig range. We are guaranteed to be holding the PTL, so any - * contiguous range cannot be unfolded or otherwise modified under our - * feet. - */ - - pte_t pte; - int i; - - ptep = contpte_align_down(ptep); - - for (i = 0; i < CONT_PTES; i++, ptep++) { - pte = __ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} -EXPORT_SYMBOL_GPL(contpte_ptep_get); - pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) { /* @@ -214,7 +181,7 @@ pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) return orig_pte; orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); - ptep = contpte_align_down(orig_ptep); + ptep = arch_contpte_align_down(orig_ptep); pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { @@ -312,7 +279,7 @@ int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, int young = 0; int i; - ptep = contpte_align_down(ptep); + ptep = arch_contpte_align_down(ptep); addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) @@ -389,7 +356,7 @@ int contpte_ptep_set_access_flags(struct vm_area_struct *vma, * faults. Avoid per-page tlb flush in __ptep_set_access_flags() * and instead flush the whole range at the end. */ - ptep = contpte_align_down(ptep); + ptep = arch_contpte_align_down(ptep); start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 5869f20ca28e..083e80ac5790 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -101,12 +101,14 @@ int find_num_contig(struct mm_struct *mm, unsigned long addr, pud_t *pudp; pmd_t *pmdp; - *pgsize = PAGE_SIZE; + if (pgsize) + *pgsize = PAGE_SIZE; p4dp = p4d_offset(pgdp, addr); pudp = pud_offset(p4dp, addr); pmdp = pmd_offset(pudp, addr); if ((pte_t *)pmdp == ptep) { - *pgsize = PMD_SIZE; + if (pgsize) + *pgsize = PMD_SIZE; return CONT_PMDS; } return CONT_PTES; diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h index 7388edd88986..f303fef8591c 100644 --- a/arch/riscv/include/asm/kfence.h +++ b/arch/riscv/include/asm/kfence.h @@ -18,9 +18,9 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) pte_t *pte = virt_to_kpte(addr); if (protect) - set_pte(pte, __pte(pte_val(ptep_get(pte)) & ~_PAGE_PRESENT)); + set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT)); else - set_pte(pte, __pte(pte_val(ptep_get(pte)) | _PAGE_PRESENT)); + set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT)); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 9e397935536e..8d05179f6bbe 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -299,6 +299,7 @@ static inline unsigned long pte_napot(pte_t pte) #define pte_cont pte_napot #define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) +#define pte_valid_cont pte_valid_napot static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { @@ -571,6 +572,17 @@ static inline int arch_contpte_get_num_contig(struct mm_struct *mm, return size >> hugepage_shift; } + +static inline pte_t *arch_contpte_align_down(pte_t *ptep) +{ + pte_t __pte = READ_ONCE(*ptep); + int ncontig; + + ncontig = napot_pte_num(napot_cont_order(__pte)); + + return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * ncontig); +} + #endif static inline pte_t __ptep_get(pte_t *ptep) @@ -696,8 +708,18 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return ptep_test_and_clear_young(vma, address, ptep); } +#ifdef CONFIG_THP_CONTPTE + +extern pte_t ptep_get(pte_t *ptep); +#define ptep_get ptep_get + +#else /* CONFIG_THP_CONTPTE */ + #define ptep_get __ptep_get #define set_ptes __set_ptes + +#endif /* CONFIG_THP_CONTPTE */ + #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear #define pte_clear __pte_clear diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c index b64bf1624a05..3d2a635c69ac 100644 --- a/arch/riscv/kernel/efi.c +++ b/arch/riscv/kernel/efi.c @@ -60,7 +60,7 @@ int __init efi_create_mapping(struct mm_struct *mm, efi_memory_desc_t *md) static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) { efi_memory_desc_t *md = data; - pte_t pte = ptep_get(ptep); + pte_t pte = __ptep_get(ptep); unsigned long val; if (md->attribute & EFI_MEMORY_RO) { diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index a9e2fd7245e1..70c6cb3864d6 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -103,7 +103,7 @@ static bool gstage_get_leaf_entry(struct kvm *kvm, gpa_t addr, *ptep_level = current_level; ptep = (pte_t *)kvm->arch.pgd; ptep = &ptep[gstage_pte_index(addr, current_level)]; - while (ptep && pte_val(ptep_get(ptep))) { + while (ptep && pte_val(__ptep_get(ptep))) { if (gstage_pte_leaf(ptep)) { *ptep_level = current_level; *ptepp = ptep; @@ -113,7 +113,7 @@ static bool gstage_get_leaf_entry(struct kvm *kvm, gpa_t addr, if (current_level) { current_level--; *ptep_level = current_level; - ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep)); ptep = &ptep[gstage_pte_index(addr, current_level)]; } else { ptep = NULL; @@ -149,7 +149,7 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, if (gstage_pte_leaf(ptep)) return -EEXIST; - if (!pte_val(ptep_get(ptep))) { + if (!pte_val(__ptep_get(ptep))) { if (!pcache) return -ENOMEM; next_ptep = kvm_mmu_memory_cache_alloc(pcache); @@ -160,7 +160,7 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, } else { if (gstage_pte_leaf(ptep)) return -EEXIST; - next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep)); } current_level--; @@ -239,11 +239,11 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, BUG_ON(addr & (page_size - 1)); - if (!pte_val(ptep_get(ptep))) + if (!pte_val(__ptep_get(ptep))) return; if (ptep_level && !gstage_pte_leaf(ptep)) { - next_ptep = (pte_t *)gstage_pte_page_vaddr(ptep_get(ptep)); + next_ptep = (pte_t *)gstage_pte_page_vaddr(__ptep_get(ptep)); next_ptep_level = ptep_level - 1; ret = gstage_level_to_page_size(next_ptep_level, &next_page_size); @@ -261,7 +261,7 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, if (op == GSTAGE_OP_CLEAR) set_pte(ptep, __pte(0)); else if (op == GSTAGE_OP_WP) - set_pte(ptep, __pte(pte_val(ptep_get(ptep)) & ~_PAGE_WRITE)); + set_pte(ptep, __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE)); gstage_remote_tlb_flush(kvm, ptep_level, addr); } } @@ -603,7 +603,7 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) &ptep, &ptep_level)) return false; - return pte_young(ptep_get(ptep)); + return pte_young(__ptep_get(ptep)); } int kvm_riscv_gstage_map(struct kvm_vcpu *vcpu, diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 3ba1d4dde5dd..0e08afc1fc6a 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -175,7 +175,7 @@ static inline void vmalloc_fault(struct pt_regs *regs, int code, unsigned long a * silently loop forever. */ pte_k = pte_offset_kernel(pmd_k, addr); - if (!pte_present(ptep_get(pte_k))) { + if (!pte_present(__ptep_get(pte_k))) { no_context(regs, addr); return; } diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index c301c8d291d2..381d61f42ab8 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -39,7 +39,7 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned ptep = pte_offset_kernel(pmd, vaddr); do { - if (pte_none(ptep_get(ptep))) { + if (pte_none(__ptep_get(ptep))) { phys_addr = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL)); memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE); diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index 410056a50aa9..98c9dc4b983c 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -68,7 +68,7 @@ static int pageattr_pmd_entry(pmd_t *pmd, unsigned long addr, static int pageattr_pte_entry(pte_t *pte, unsigned long addr, unsigned long next, struct mm_walk *walk) { - pte_t val = ptep_get(pte); + pte_t val = __ptep_get(pte); val = __pte(set_pageattr_masks(pte_val(val), walk)); set_pte(pte, val); @@ -435,5 +435,5 @@ bool kernel_page_present(struct page *page) return true; pte = pte_offset_kernel(pmd, addr); - return pte_present(ptep_get(pte)); + return pte_present(__ptep_get(pte)); } diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index e86df7ef193c..5756bde9eb42 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -9,7 +9,7 @@ int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) { - if (!pte_same(ptep_get(ptep), entry)) + if (!pte_same(__ptep_get(ptep), entry)) __set_pte_at(vma->vm_mm, ptep, entry); /* * update_mmu_cache will unconditionally execute, handling both @@ -22,7 +22,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (!pte_young(ptep_get(ptep))) + if (!pte_young(__ptep_get(ptep))) return 0; return test_and_clear_bit(_PAGE_ACCESSED_OFFSET, &pte_val(*ptep)); } diff --git a/include/linux/contpte.h b/include/linux/contpte.h new file mode 100644 index 000000000000..46acac7222ca --- /dev/null +++ b/include/linux/contpte.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_CONTPTE_H +#define _LINUX_CONTPTE_H + +/* + * The contpte APIs are used to transparently manage the contiguous bit in ptes + * where it is possible and makes sense to do so. The PTE_CONT bit is considered + * a private implementation detail of the public ptep API (see below). + */ +pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); + +#endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 15791f6d9c41..d365356bbf92 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -6,6 +6,7 @@ #include #include #include +#include /* * Any arch that wants to use that needs to define: @@ -17,6 +18,8 @@ * - __ptep_set_wrprotect() * - pte_cont() * - arch_contpte_get_num_contig() + * - pte_valid_cont() + * - arch_contpte_align_down() */ /* @@ -28,6 +31,7 @@ * - huge_ptep_set_access_flags() * - huge_ptep_set_wrprotect() * - huge_ptep_clear_flush() + * - ptep_get() */ pte_t huge_ptep_get(pte_t *ptep) @@ -270,3 +274,44 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); } + +#ifdef CONFIG_THP_CONTPTE +pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) +{ + /* + * Gather access/dirty bits, which may be populated in any of the ptes + * of the contig range. We are guaranteed to be holding the PTL, so any + * contiguous range cannot be unfolded or otherwise modified under our + * feet. + */ + + pte_t pte; + int i, ncontig; + + ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(NULL, 0, ptep, 0, NULL); + + for (i = 0; i < ncontig; i++, ptep++) { + pte = __ptep_get(ptep); + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} +EXPORT_SYMBOL_GPL(contpte_ptep_get); + +__always_inline pte_t ptep_get(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(pte))) + return pte; + + return contpte_ptep_get(ptep, pte); +} +#endif /* CONTPTE_THP_CONTPTE */ From patchwork Wed May 8 19:19:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 836DAC25B4F for ; Wed, 8 May 2024 19:22:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CF1E6B0098; Wed, 8 May 2024 15:22:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 180236B0099; Wed, 8 May 2024 15:22:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F3BC26B009A; Wed, 8 May 2024 15:22:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D399C6B0098 for ; Wed, 8 May 2024 15:22:42 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8A40740EBE for ; Wed, 8 May 2024 19:22:42 +0000 (UTC) X-FDA: 82096200564.10.0EDDD53 Received: from mail-wm1-f44.google.com (mail-wm1-f44.google.com [209.85.128.44]) by imf22.hostedemail.com (Postfix) with ESMTP id 7D10BC0003 for ; Wed, 8 May 2024 19:22:40 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=WdtE9l13; dmarc=none; spf=pass (imf22.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.44 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196160; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/S8/CVFlZq+KrirwWJV8BSI4dNCVaFyTIK5dU7sIVgI=; b=L29/66LdTejoGKHxoZ59cHaJocSMSBm9EzPPpyjuMjSznUrRFtcNgbEaOFj/62N7vgXJ+h sP6/M4nvCcDS3bZbb7jkc1q9tjBUP2ohdOWduXy7SmY5Hp7p1+WWgz0obDGxZWD54sTcx+ 9mA/+qMcCHDlXTG1hW6OF8M0ni0FEdI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=WdtE9l13; dmarc=none; spf=pass (imf22.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.44 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196160; a=rsa-sha256; cv=none; b=54zcji8ClIuNzKLv9GEwV4oQNXYBsR2Amy6v74oavSLfXwXcjz3qQc6yDkF6NHjcFPSIgi of2Hl81kd/acIqyc6qdVN1q9FXa9ujIIow4xjemnEUerno7WHmU500hTKksWJi4hLynl/d 2XVW/P27xrPF4tAP4GvClHkT+ifd9w0= Received: by mail-wm1-f44.google.com with SMTP id 5b1f17b1804b1-41ebcf01013so850665e9.0 for ; Wed, 08 May 2024 12:22:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196159; x=1715800959; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/S8/CVFlZq+KrirwWJV8BSI4dNCVaFyTIK5dU7sIVgI=; b=WdtE9l13nYMv1P+btNiSpdUf/8iqdUoYOPQIGINtFHnbJrxBVweFG7mZ/PnRkr2kwZ EoKZvtwNHv9kwbQtgKn3Eo85eW/Gbx/VLnFzOlXaxapIqwSqxUXvZH388mKiAAkmcwLD Jtgl1wIdmOTms/PCUpkqwdKGi8LifaLcsNMuzMMpeNjiefCbzGSp0DFnhFF+UFTBwqxQ 65jET6HqWlS0c9DPgGz+pQY40wW4ScowPfWfxM37HwI0G6l7zQRSQgsytdfS+xNSSNrh ssX1vXrmJ0a+WGVyeP4RymSYxW8USUt1r3oW1o5AoFej+iApD7HGMsxOeWArAijeZhW/ 1GrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196159; x=1715800959; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/S8/CVFlZq+KrirwWJV8BSI4dNCVaFyTIK5dU7sIVgI=; b=Z9hxsHLhOoM3iVDzBnq2dkFwA4rj/CJrc/6YUFo34fV/CLt0ShVV2j5ZyG49hsejYh l7MwxkaVhQcRjB/BCH9MLc/CVtjSfKkhg+WtYz31xbEKhdrXM/rzhYqP3RxDxWD81A8r AxinTYzupe1Xz0I8SC4A2cQgmab99mZmUi/dQWNIaVN5MrEZqxQOUv3FrHEDRy62d/2u Zz8U/LtttN6XAJRfBxHYu5+KoOO+SvkIRfa2RW1tj9+d13m0H696f0LZmocwGy2KGSwR CQv3w9u3MUS9XQ9FyNu+yzuPxDzqeHBPS9qN75V9gsT1HQhj3vX3W+yniaXCE9FzxDUJ felw== X-Forwarded-Encrypted: i=1; AJvYcCVzTrT/WaxANReYlclXBjAnopHYPTyZBb/mB9XieJn2epxVumkvQCVxE5lkOwCa8x6pY8NBBAYknkDuWQPI4s8mvpw= X-Gm-Message-State: AOJu0YxhXKGkvqBrNgu+9wSWcjAZtDAXMjrNflM+tXeysjUk91S0O8AD 0KMFZuuPxMCXMbSqPkkMx7s6nUxE6AbtqxQN5ayGCWjh6tY2uA795DWJ/Npnles= X-Google-Smtp-Source: AGHT+IGv6186BLCsPjXiNHxvqIGGbP+8Nt6Ojq3omkEdJDigbcQHRKHJSnceRKhu854h+/CEdBxYBA== X-Received: by 2002:a05:600c:3b83:b0:41b:4506:9fd with SMTP id 5b1f17b1804b1-41fbcb42dd6mr4932615e9.6.1715196158601; Wed, 08 May 2024 12:22:38 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id f20-20020a05600c155400b0041bf7da4200sm3310028wmg.33.2024.05.08.12.22.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:22:38 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 03/12] mm, riscv, arm64: Use common set_ptes() function Date: Wed, 8 May 2024 21:19:22 +0200 Message-Id: <20240508191931.46060-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Stat-Signature: ks3jpj81qnnsmkjsxezo37prnhghkeqe X-Rspamd-Queue-Id: 7D10BC0003 X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1715196160-575436 X-HE-Meta: U2FsdGVkX18f/WGbNwk+xShWZRqMQfmKtzQ/lVgGzQqWyzhPrhc/IXzl1hD43qRAfQFVPNFDqTQknOTQis8MCwNl/NHJmVdmH8Ig0Kl7VCklQazdPHt0caBJYDthurh/zoB3IhbRHtCvL4/0cm12zrVeshwHwzPvJo75w+lqEQN9jtfbxSLe+/Zrnrqa8AmkDUgWdf8ihTKLCQqXQphL1iVKcqt5JjTVS43goLc138qwvxrO3tMliVNdi6xl7bU+ts94jNi531PL2+F5jEwv+Dhw5Esf2KKT8+c2oIoboUAD7ubllH2QWPGWsR2t1ueSNflEyAp/cggU8Pn4bFIWysk0+3nErKmOJ21jFznydADKVShxELhxw74T1bQNPkaP9BEPQ0Joqkt2zld8VERI7GhHzBos1wp0YZQGiY0iwWTxaHi/gKzXdwnoaaW3zH9WRN8NKSgH+37zGHb2WADOCGgfKlQTJ8IHu2uSUSeeRyIQTGIAnc/CWkvCT76mfTstzYwAkijMR77XjZKWfCcHSG2bLFQDC3aD6YLbNKEQPSPBbxEzxv1L14D8WG6sPNbkbYbHTfY25VyKzFwCw4hhihb4+63k5S5b5b/XPQLHPH7ult04yyY8N3yNtVP5X4y1fbK9clkx0rJKhY5095arhwpESuM/BghFW3/4rteD+br7rgHz9IxwfytE3WDmvB7CawBaAQo0u7xSqlUHr9ETJz2pZRhypRVgFXjQEtCLPHBQw9Un5A7KK7ujPuZiKlrRY64958GlDlLXfm+PW0KeThpq8pmEwhSsAfRVnfKtiN09YyMFBz56dkyq3CIu77GwE91H9VNrMZ8HvRawRMT4bGfTi1FNcUwxScyk753AWnWVVfXg4Q8HYGT/DVSFHg0CvBBJVhi28lkqSBPbSnbr2J2Vbw0EcVabHe3+34yauigB2qsN9/cVwWwKQg1l6qsf5qU2nzQRK6dhDyBYsnM LWOD0rH8 8wHShsw9FoMar5dDIgWHob6kymtvkuspGlQyM937qbZsGPMk2RwntQp985ax1ysdtHklc2UcrbFO7UZ7Sg7Mz/IXdB6xrxaCDWZa/dWR9gDlSaamwtaL48aJOVSUtXoNG0yJg+vSVmdEuL/IU79mNT5PXRZ9jRXgWcTsZ7qb6tzOftQDJKoS8vd3VBwQcENDZM20Wjm1rwDPXTpbbndn2h7sX6YoGQ5Zci1QfSclFL9hI4NhWHHUPHwg5JoQE9A4i5yKihhaTIrymHZtsNWy7xFnqgvnEJ0dbs1UL/OPCUlaSVRw3tDT5Ail1Z17ohDd5qhD3P/gZOXKhlIF5EvTipIFIaw1Ev4QXBd5KWd4oEZ5725/ufvvL/15QSzJyCzTjt97EppJVjCRUE86LU/MGBY4PRWNM39TYhzZK7oX7dXs+1XIQS+QDSuWitxBy1zrgbHw+sdL4zaunjPsptER1DlkVprDnxn9N3zM6jkuOIgvTcOloRP/vnPuu3tlRiTe/lwZsH7DH55ZPTkiTFepmp5DPbnDhVrlk/t1YggyNC2yiMKho0LmfqFPMbA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware set_ptes() function from arm64. Note that riscv can support multiple contpte sizes so the arm64 code was modified to take into account this possibility. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 88 ++++------ arch/arm64/mm/contpte.c | 162 ------------------ arch/arm64/mm/mmu.c | 2 +- arch/riscv/include/asm/pgtable.h | 76 +++++++++ include/linux/contpte.h | 10 ++ mm/contpte.c | 277 ++++++++++++++++++++++++++++++- 6 files changed, 398 insertions(+), 217 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index a878735deb9f..e85b3a052a02 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -118,10 +118,17 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) #define pte_tagged(pte) ((pte_val(pte) & PTE_ATTRINDX_MASK) == \ PTE_ATTRINDX(MT_NORMAL_TAGGED)) -#define pte_cont_addr_end(addr, end) \ -({ unsigned long __boundary = ((addr) + CONT_PTE_SIZE) & CONT_PTE_MASK; \ - (__boundary - 1 < (end) - 1) ? __boundary : (end); \ -}) +static inline unsigned long arch_contpte_addr_end(unsigned long addr, + unsigned long end, + int *ncontig) +{ + unsigned long __boundary = (addr + CONT_PTE_SIZE) & CONT_PTE_MASK; + + if (ncontig) + *ncontig = CONT_PTES; + + return (__boundary - 1 < end - 1) ? __boundary : end; +} #define pmd_cont_addr_end(addr, end) \ ({ unsigned long __boundary = ((addr) + CONT_PMD_SIZE) & CONT_PMD_MASK; \ @@ -1377,13 +1384,7 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, * where it is possible and makes sense to do so. The PTE_CONT bit is considered * a private implementation detail of the public ptep API (see below). */ -extern void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte); -extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); -extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned int nr); extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, @@ -1399,36 +1400,6 @@ extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); -static __always_inline void contpte_try_fold(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, pte_t pte) -{ - /* - * Only bother trying if both the virtual and physical addresses are - * aligned and correspond to the last entry in a contig range. The core - * code mostly modifies ranges from low to high, so this is the likely - * the last modification in the contig range, so a good time to fold. - * We can't fold special mappings, because there is no associated folio. - */ - - const unsigned long contmask = CONT_PTES - 1; - bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask; - - if (unlikely(valign)) { - bool palign = (pte_pfn(pte) & contmask) == contmask; - - if (unlikely(palign && - pte_valid(pte) && !pte_cont(pte) && !pte_special(pte))) - __contpte_try_fold(mm, addr, ptep, pte); - } -} - -static __always_inline void contpte_try_unfold(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, pte_t pte) -{ - if (unlikely(pte_valid_cont(pte))) - __contpte_try_unfold(mm, addr, ptep, pte); -} - #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) { @@ -1485,20 +1456,9 @@ static inline void set_pte(pte_t *ptep, pte_t pte) __set_pte(ptep, pte_mknoncont(pte)); } +extern void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr); #define set_ptes set_ptes -static __always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned int nr) -{ - pte = pte_mknoncont(pte); - - if (likely(nr == 1)) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __set_ptes(mm, addr, ptep, pte, 1); - contpte_try_fold(mm, addr, ptep, pte); - } else { - contpte_set_ptes(mm, addr, ptep, pte, nr); - } -} static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -1686,6 +1646,28 @@ static inline pte_t *arch_contpte_align_down(pte_t *ptep) return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * CONT_PTES); } +static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, + unsigned long end, + unsigned long stride) +{ + __flush_tlb_range(vma, start, end, stride, true, 3); +} + +static inline int arch_contpte_get_first_ncontig(size_t *pgsize) +{ + if (pgsize) + *pgsize = PAGE_SIZE; + + return CONT_PTES; +} + +/* Must return 0 when ncontig does not have any next. */ +static inline int arch_contpte_get_next_ncontig(int ncontig, size_t *pgsize) +{ + return 0; +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index d5512ebb26e9..e225e458856e 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -8,19 +8,6 @@ #include #include -static inline bool mm_is_user(struct mm_struct *mm) -{ - /* - * Don't attempt to apply the contig bit to kernel mappings, because - * dynamically adding/removing the contig bit can cause page faults. - * These racing faults are ok for user space, since they get serialized - * on the PTL. But kernel mappings can't tolerate faults. - */ - if (unlikely(mm_is_efi(mm))) - return false; - return mm != &init_mm; -} - static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) { @@ -41,112 +28,6 @@ static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, } } -static void contpte_convert(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long start_addr; - pte_t *start_ptep; - int i; - - start_ptep = ptep = arch_contpte_align_down(ptep); - start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte)); - - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) { - pte_t ptent = __ptep_get_and_clear(mm, addr, ptep); - - if (pte_dirty(ptent)) - pte = pte_mkdirty(pte); - - if (pte_young(ptent)) - pte = pte_mkyoung(pte); - } - - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); - - __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); -} - -void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) -{ - /* - * We have already checked that the virtual and pysical addresses are - * correctly aligned for a contpte mapping in contpte_try_fold() so the - * remaining checks are to ensure that the contpte range is fully - * covered by a single folio, and ensure that all the ptes are valid - * with contiguous PFNs and matching prots. We ignore the state of the - * access and dirty bits for the purpose of deciding if its a contiguous - * range; the folding process will generate a single contpte entry which - * has a single access and dirty bit. Those 2 bits are the logical OR of - * their respective bits in the constituent pte entries. In order to - * ensure the contpte range is covered by a single folio, we must - * recover the folio from the pfn, but special mappings don't have a - * folio backing them. Fortunately contpte_try_fold() already checked - * that the pte is not special - we never try to fold special mappings. - * Note we can't use vm_normal_page() for this since we don't have the - * vma. - */ - - unsigned long folio_start, folio_end; - unsigned long cont_start, cont_end; - pte_t expected_pte, subpte; - struct folio *folio; - struct page *page; - unsigned long pfn; - pte_t *orig_ptep; - pgprot_t prot; - - int i; - - if (!mm_is_user(mm)) - return; - - page = pte_page(pte); - folio = page_folio(page); - folio_start = addr - (page - &folio->page) * PAGE_SIZE; - folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE; - cont_start = ALIGN_DOWN(addr, CONT_PTE_SIZE); - cont_end = cont_start + CONT_PTE_SIZE; - - if (folio_start > cont_start || folio_end < cont_end) - return; - - pfn = ALIGN_DOWN(pte_pfn(pte), CONT_PTES); - prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); - expected_pte = pfn_pte(pfn, prot); - orig_ptep = ptep; - ptep = arch_contpte_align_down(ptep); - - for (i = 0; i < CONT_PTES; i++) { - subpte = pte_mkold(pte_mkclean(__ptep_get(ptep))); - if (!pte_same(subpte, expected_pte)) - return; - expected_pte = pte_advance_pfn(expected_pte, 1); - ptep++; - } - - pte = pte_mkcont(pte); - contpte_convert(mm, addr, orig_ptep, pte); -} -EXPORT_SYMBOL_GPL(__contpte_try_fold); - -void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) -{ - /* - * We have already checked that the ptes are contiguous in - * contpte_try_unfold(), so just check that the mm is user space. - */ - if (!mm_is_user(mm)) - return; - - pte = pte_mknoncont(pte); - contpte_convert(mm, addr, ptep, pte); -} -EXPORT_SYMBOL_GPL(__contpte_try_unfold); - pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) { /* @@ -204,49 +85,6 @@ pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) } EXPORT_SYMBOL_GPL(contpte_ptep_get_lockless); -void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned int nr) -{ - unsigned long next; - unsigned long end; - unsigned long pfn; - pgprot_t prot; - - /* - * The set_ptes() spec guarantees that when nr > 1, the initial state of - * all ptes is not-present. Therefore we never need to unfold or - * otherwise invalidate a range before we set the new ptes. - * contpte_set_ptes() should never be called for nr < 2. - */ - VM_WARN_ON(nr == 1); - - if (!mm_is_user(mm)) - return __set_ptes(mm, addr, ptep, pte, nr); - - end = addr + (nr << PAGE_SHIFT); - pfn = pte_pfn(pte); - prot = pte_pgprot(pte); - - do { - next = pte_cont_addr_end(addr, end); - nr = (next - addr) >> PAGE_SHIFT; - pte = pfn_pte(pfn, prot); - - if (((addr | next | (pfn << PAGE_SHIFT)) & ~CONT_PTE_MASK) == 0) - pte = pte_mkcont(pte); - else - pte = pte_mknoncont(pte); - - __set_ptes(mm, addr, ptep, pte, nr); - - addr = next; - ptep += nr; - pfn += nr; - - } while (addr != end); -} -EXPORT_SYMBOL_GPL(contpte_set_ptes); - void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full) { diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 495b732d5af3..b7ad732660aa 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -222,7 +222,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, do { pgprot_t __prot = prot; - next = pte_cont_addr_end(addr, end); + next = arch_contpte_addr_end(addr, end, NULL); /* use a contiguous mapping if the range is suitably aligned */ if ((((addr | next | phys) & ~CONT_PTE_MASK) == 0) && diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 8d05179f6bbe..ebfe6b16529e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -301,6 +301,20 @@ static inline unsigned long pte_napot(pte_t pte) #define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) #define pte_valid_cont pte_valid_napot +/* + * contpte is what we expose to the core mm code, this is not exactly a napot + * mapping since the size is not encoded in the pfn yet. + */ +static inline pte_t pte_mknoncont(pte_t pte) +{ + return __pte(pte_val(pte) & ~_PAGE_NAPOT); +} + +static inline pte_t pte_mkcont(pte_t pte) +{ + return __pte(pte_val(pte) | _PAGE_NAPOT); +} + static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { int pos = order - 1 + _PAGE_PFN_SHIFT; @@ -329,6 +343,11 @@ static inline unsigned long pte_napot(pte_t pte) #endif /* CONFIG_RISCV_ISA_SVNAPOT */ +static inline pgprot_t pte_pgprot(pte_t pte) +{ + return __pgprot(pte_val(pte) & ~_PAGE_PFN_MASK); +} + /* Yields the page frame number (PFN) of a page table entry */ static inline unsigned long pte_pfn(pte_t pte) { @@ -354,6 +373,11 @@ static inline int pte_present(pte_t pte) return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)); } +static inline int pte_valid(pte_t pte) +{ + return (pte_val(pte) & _PAGE_PRESENT); +} + static inline int pte_none(pte_t pte) { return (pte_val(pte) == 0); @@ -583,6 +607,55 @@ static inline pte_t *arch_contpte_align_down(pte_t *ptep) return PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * ncontig); } +static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, + unsigned long end, + unsigned long stride) +{ + flush_tlb_mm_range(vma->vm_mm, start, end, stride); +} + +static inline int arch_contpte_get_first_ncontig(size_t *pgsize) +{ + if (pgsize) + *pgsize = PAGE_SIZE; + + return 1 << NAPOT_CONT64KB_ORDER; +} + +/* Must return 0 when ncontig does not have any next. */ +static inline int arch_contpte_get_next_ncontig(int ncontig, size_t *pgsize) +{ + return 0; +} + +#define for_each_contpte_order_rev(ncontig, order, pgsize) \ + for (pgsize = PAGE_SIZE, order = NAPOT_ORDER_MAX - 1, ncontig = BIT(order); \ + ncontig >= BIT(NAPOT_CONT_ORDER_BASE); \ + order--, ncontig = BIT(order)) + +static inline unsigned long arch_contpte_addr_end(unsigned long addr, + unsigned long end, + int *ncontig) +{ + unsigned long contpte_saddr, contpte_eaddr, contpte_size; + size_t pgsize; + int contig, order; + + for_each_contpte_order_rev(contig, order, pgsize) { + contpte_size = contig * pgsize; + contpte_saddr = ALIGN_DOWN(addr, contpte_size); + contpte_eaddr = contpte_saddr + contpte_size; + + if (contpte_saddr >= addr && contpte_eaddr <= end) { + *ncontig = contig; + return contpte_eaddr; + } + } + + *ncontig = 0; + return end; +} #endif static inline pte_t __ptep_get(pte_t *ptep) @@ -712,6 +785,9 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, extern pte_t ptep_get(pte_t *ptep); #define ptep_get ptep_get +extern void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pteval, unsigned int nr); +#define set_ptes set_ptes #else /* CONFIG_THP_CONTPTE */ diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 46acac7222ca..54d10204e9af 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -8,5 +8,15 @@ * a private implementation detail of the public ptep API (see below). */ pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); +void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index d365356bbf92..566745d7842f 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -7,6 +7,7 @@ #include #include #include +#include /* * Any arch that wants to use that needs to define: @@ -20,6 +21,14 @@ * - arch_contpte_get_num_contig() * - pte_valid_cont() * - arch_contpte_align_down() + * - arch_contpte_flush_tlb_range() + * - arch_contpte_get_first_ncontig() + * - arch_contpte_get_next_ncontig() + * - arch_contpte_addr_end() + * - pte_mkcont() + * - pte_mknoncont() + * - pte_valid() + * - pte_pgprot() */ /* @@ -32,6 +41,7 @@ * - huge_ptep_set_wrprotect() * - huge_ptep_clear_flush() * - ptep_get() + * - set_ptes() */ pte_t huge_ptep_get(pte_t *ptep) @@ -314,4 +324,269 @@ __always_inline pte_t ptep_get(pte_t *ptep) return contpte_ptep_get(ptep, pte); } -#endif /* CONTPTE_THP_CONTPTE */ +EXPORT_SYMBOL_GPL(ptep_get); + +static inline bool mm_is_user(struct mm_struct *mm) +{ + /* + * Don't attempt to apply the contig bit to kernel mappings, because + * dynamically adding/removing the contig bit can cause page faults. + * These racing faults are ok for user space, since they get serialized + * on the PTL. But kernel mappings can't tolerate faults. + */ + if (unlikely(mm_is_efi(mm))) + return false; + return mm != &init_mm; +} + +static void contpte_convert(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, + int ncontig, size_t pgsize) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long start_addr; + pte_t *start_ptep; + int i; + + start_addr = addr; + start_ptep = ptep; + + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) { + pte_t ptent = __ptep_get_and_clear(mm, addr, ptep); + + if (pte_dirty(ptent)) + pte = pte_mkdirty(pte); + + if (pte_young(ptent)) + pte = pte_mkyoung(pte); + } + + arch_contpte_flush_tlb_range(&vma, start_addr, addr, pgsize); + + __set_ptes(mm, start_addr, start_ptep, pte, ncontig); +} + +void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + unsigned long start_addr; + pte_t *start_ptep; + size_t pgsize; + int ncontig; + + /* + * We have already checked that the ptes are contiguous in + * contpte_try_unfold(), so just check that the mm is user space. + */ + if (!mm_is_user(mm)) + return; + + pte = pte_mknoncont(pte); + start_ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(mm, addr, start_ptep, 0, &pgsize); + start_addr = ALIGN_DOWN(addr, ncontig * pgsize); + pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), ncontig), pte_pgprot(pte)); + + contpte_convert(mm, start_addr, start_ptep, pte, ncontig, pgsize); +} + +__always_inline void contpte_try_unfold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + pte_t pte) +{ + if (unlikely(pte_valid_cont(pte))) + __contpte_try_unfold(mm, addr, ptep, pte); +} +EXPORT_SYMBOL_GPL(contpte_try_unfold); + +static bool contpte_is_last_pte(unsigned long addr, pte_t pte, int ncontig) +{ + const unsigned long contmask = ncontig - 1; + bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask; + + if (unlikely(valign)) { + bool palign = (pte_pfn(pte) & contmask) == contmask; + + if (unlikely(palign && + pte_valid(pte) && !pte_cont(pte) && !pte_special(pte))) + return true; + } + + return false; +} + +void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the virtual and pysical addresses are + * correctly aligned for a contpte mapping in contpte_try_fold() so the + * remaining checks are to ensure that the contpte range is fully + * covered by a single folio, and ensure that all the ptes are valid + * with contiguous PFNs and matching prots. We ignore the state of the + * access and dirty bits for the purpose of deciding if its a contiguous + * range; the folding process will generate a single contpte entry which + * has a single access and dirty bit. Those 2 bits are the logical OR of + * their respective bits in the constituent pte entries. In order to + * ensure the contpte range is covered by a single folio, we must + * recover the folio from the pfn, but special mappings don't have a + * folio backing them. Fortunately contpte_try_fold() already checked + * that the pte is not special - we never try to fold special mappings. + * Note we can't use vm_normal_page() for this since we don't have the + * vma. + */ + + unsigned long folio_start, folio_end; + unsigned long cont_start, cont_end; + pte_t expected_pte, subpte; + struct folio *folio; + struct page *page; + unsigned long pfn; + pte_t *cur_ptep; + pgprot_t prot; + size_t pgsize; + int i, ncontig, ncontig_prev; + + if (!mm_is_user(mm)) + return; + + page = pte_page(pte); + folio = page_folio(page); + folio_start = addr - (page - &folio->page) * PAGE_SIZE; + folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE; + + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + ncontig_prev = 0; + ncontig = arch_contpte_get_first_ncontig(&pgsize); + + do { + /* Make sure we still belong to the same folio */ + cont_start = ALIGN_DOWN(addr, ncontig * pgsize); + cont_end = cont_start + ncontig * pgsize; + + if (folio_start > cont_start || folio_end < cont_end) + break; + + pfn = ALIGN_DOWN(pte_pfn(pte), ncontig); + expected_pte = pfn_pte(pfn, prot); + cur_ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * ncontig); + + /* + * Either the region is already a contpte mapping or make sure + * the ptes belong to a contpte region. + */ + if (!pte_valid_cont(__ptep_get(cur_ptep))) { + for (i = 0; i < ncontig - ncontig_prev; i++) { + subpte = pte_mkold(pte_mkclean(__ptep_get(cur_ptep))); + if (!pte_same(subpte, expected_pte)) + goto out; + expected_pte = pte_advance_pfn(expected_pte, 1); + cur_ptep++; + } + } + + /* + * Compute the next contpte region to check: avoid checking + * the same region again by only checking the "second-half" + * (the "region buddy") of the current region, which keeps the + * whole thing in O(n). + */ + ncontig_prev = ncontig; + ncontig = arch_contpte_get_next_ncontig(ncontig, &pgsize); + /* set_ptes spec states that "The PTEs are all in the same PMD" */ + if (!ncontig || pgsize > PAGE_SIZE) + break; + } while (contpte_is_last_pte(addr, pte, ncontig)); + +out: + if (!ncontig_prev) + return; + + ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * ncontig_prev); + cont_start = ALIGN_DOWN(addr, ncontig_prev * PAGE_SIZE); + pte = pte_mkcont(pte); + pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), ncontig_prev), pte_pgprot(pte)); + + contpte_convert(mm, cont_start, ptep, pte, ncontig_prev, PAGE_SIZE); +} +EXPORT_SYMBOL_GPL(__contpte_try_fold); + +__always_inline void contpte_try_fold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte) +{ + /* + * Only bother trying if both the virtual and physical addresses are + * aligned and correspond to the last entry in a contig range. The core + * code mostly modifies ranges from low to high, so this is the likely + * the last modification in the contig range, so a good time to fold. + * We can't fold special mappings, because there is no associated folio. + * + * Only test if the first ncontig is aligned, because it can't be the + * last pte of a size larger than the first ncontig and not the last + * of the first ncontig. + */ + int ncontig = arch_contpte_get_first_ncontig(NULL); + + if (contpte_is_last_pte(addr, pte, ncontig)) + __contpte_try_fold(mm, addr, ptep, pte); +} + +void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + unsigned long contmask; + unsigned long next; + unsigned long end; + unsigned long pfn; + pgprot_t prot; + int ncontig; + + /* + * The set_ptes() spec guarantees that when nr > 1, the initial state of + * all ptes is not-present. Therefore we never need to unfold or + * otherwise invalidate a range before we set the new ptes. + * contpte_set_ptes() should never be called for nr < 2. + */ + VM_WARN_ON(nr == 1); + + if (!mm_is_user(mm)) + return __set_ptes(mm, addr, ptep, pte, nr); + + end = addr + (nr << PAGE_SHIFT); + pfn = pte_pfn(pte); + prot = pte_pgprot(pte); + + do { + next = arch_contpte_addr_end(addr, end, &ncontig); + nr = (next - addr) >> PAGE_SHIFT; + pte = pfn_pte(pfn, prot); + contmask = (ncontig << PAGE_SHIFT) - 1; + + if (ncontig && ((addr | next | (pfn << PAGE_SHIFT)) & contmask) == 0) + pte = pte_mkcont(pte); + else + pte = pte_mknoncont(pte); + + __set_ptes(mm, addr, ptep, pte, nr); + + addr = next; + ptep += nr; + pfn += nr; + } while (addr != end); +} +EXPORT_SYMBOL_GPL(contpte_set_ptes); + +__always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + pte = pte_mknoncont(pte); + + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __set_ptes(mm, addr, ptep, pte, 1); + contpte_try_fold(mm, addr, ptep, pte); + } else { + contpte_set_ptes(mm, addr, ptep, pte, nr); + } +} +#endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6397AC04FFE for ; Wed, 8 May 2024 19:23:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F16476B009A; Wed, 8 May 2024 15:23:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC6966B009B; Wed, 8 May 2024 15:23:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D67526B009C; Wed, 8 May 2024 15:23:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B7B496B009A for ; Wed, 8 May 2024 15:23:43 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6DDF8806CF for ; Wed, 8 May 2024 19:23:43 +0000 (UTC) X-FDA: 82096203126.21.2B89EA4 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by imf24.hostedemail.com (Postfix) with ESMTP id 8563E180013 for ; Wed, 8 May 2024 19:23:41 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=0fTpowZu; dmarc=none; spf=pass (imf24.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196221; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=99tlcIPBJBGOnw31Oe30kKMBjt8TwvO6BCavKhZeDaw=; b=XsxXIpqIsy7AdZLuP48Y7DGdn23Pn4EN81cKZsXy+v1pkGVQ7V9UHZLGIaUaECHPlHcx9O SXpK518qJoGYuK3n3w8WcjQK8n6blJ9A3nhacVDDL9+cDtgPvJAXd1PXnfnXOxBKcBVTw4 dGVP1Yw7/XAsVdHFE8VIPKx40ve3egg= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=0fTpowZu; dmarc=none; spf=pass (imf24.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.45 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196221; a=rsa-sha256; cv=none; b=cfCC6Bzh69bqdIZ0quc9q7XO25DwwQA6oIr6wsu85mSlfxLKAh/+MS3Q3VjQL78ki9H2Q6 YEmHeL6vokgG/z916iOrClsX5z1m+pTwb3AoOCYMy8cUv7o1mi/2sAqCEommMSZ7s4ClWr r9zF/c5/52INFG0HqHliS/NhEKTeX3Y= Received: by mail-wm1-f45.google.com with SMTP id 5b1f17b1804b1-41fc2f7fbb5so612745e9.1 for ; Wed, 08 May 2024 12:23:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196220; x=1715801020; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=99tlcIPBJBGOnw31Oe30kKMBjt8TwvO6BCavKhZeDaw=; b=0fTpowZu63nTPD5lUf9AzB6IA1IdwkvBqSpTvrgE/k+axPzZ0wrEVjd63xX99y9wUR gJG6UI5Fe4lS7pP0thGL4dhUqlt4PhojACh+pLsQgFpKJo11HQ+NLXVRXqj1hcVGocjp tvZ3+CmmWXSEmp2UO9+4oJ5oIB1w+WIZI9Z6RMKGKQSvavfdjdA9Rsej+pkZF8Mg/oXN vio7AfOU/LYz09Y83PA3Gl4cCBuf1P3T1gyX1Q++me4WHT6uED/9uer6SlEiRA2Uqy8p /fi4doVYaMdoZ1hRDiyu5jBjpxC5tjMKd+wdSRnRNhhb3Ii2s6aEo3H1xFGXOsnz7ycP 0m8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196220; x=1715801020; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=99tlcIPBJBGOnw31Oe30kKMBjt8TwvO6BCavKhZeDaw=; b=QZP0GNiQO/GjJKFtDBqxTCDxpmX9Eur3O8zMsFUzoPk6PHUmsVW7Ke5PcMEIoQ+Bhm Wri+FsCJE4asv9ih9p+eBhia9sFO0txdzzDyIBHG6eXPVIsmkRqmO0snmeySlQAewfEl Jdp5F5AnK+TQ1aWP3DfksW6RlzL51+ba5VNQ+P4epe1rYWMMeRMYDYYj/BmuQqeB/ND4 G7J5wVtJRfBAWSa6zy5Ck+/wAY7iwfWyBC5/iYsHvJCVQ+b62jPS18dNtVCp7Up2DFdr zzxb+M35KF+ffjj7nTzzXLysBA5UVI03d+FglnZYYCwoznKZtrlJg5jKntveOBrSiL13 iRnA== X-Forwarded-Encrypted: i=1; AJvYcCXHNi0+G+QXnCWuSYxj/deyDw+yyjtRmJZOz7Tf9RdaMSvqf3ttoV67IqMFCLpD39pl+bGEIX/urgdosOQqHRpvZkE= X-Gm-Message-State: AOJu0YyibI04VLH2a17ciI+1Uq0c6eSVTroZ1MzBQWnVn+8veQsq+ymI O//DmFnEeC90UU5fGTQqev+v9WLIQJ1sn9KSysILBUOzHMrmP2qX02Km8ytHStI= X-Google-Smtp-Source: AGHT+IFuqFkdx1GrRMX9VnSrFsQl595h33hB5u187da/mHSb1Bien42UkrHvFd9ITnGVH1r4SnsHfw== X-Received: by 2002:a05:600c:3ca9:b0:41b:ed36:e055 with SMTP id 5b1f17b1804b1-41fbcb4b4fbmr4761445e9.7.1715196220131; Wed, 08 May 2024 12:23:40 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-41f42e74625sm47255965e9.0.2024.05.08.12.23.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:23:39 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 04/12] mm, riscv, arm64: Use common ptep_get_lockless() function Date: Wed, 8 May 2024 21:19:23 +0200 Message-Id: <20240508191931.46060-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Stat-Signature: rtea3gdne7dr6tgg9teziej68ghqy1dd X-Rspamd-Queue-Id: 8563E180013 X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1715196221-478852 X-HE-Meta: U2FsdGVkX1+nc7nbCYI7US5aFGJIue/Gc7HCQtAnY/JkPPaWEGWHumQQLy+fphh6LteanVRrzfORQKItbS/eRaF2nlL+hmEGQu3toptYQkhWJnmca/+sRO/lyZc+53no2JmzsWEuGD9xPUM26X5jarWk/DvMWM8FEbmBo3eYitrK3z/ce5OvvsPj0ri5YCTROIIUCupba5HcI25p/kya8WViG4KLHl/ZIbjhpQ5jPPTrNxvtf1VIi3D7fikzVpeiI9/aa2LOBEsNu2TQSkC1m7ct2X1HJKhA2vgy3FDnqiTiws+2A8VkLOq5jhiDC0IfJbjbCaRJdCjtc7dW1wPL1ei5WRrsPPF77B2e9T4XFeiEjCTjCpVNHOwEw5hGXhrsmE7HWCQv+LuiU0A1nG1+HtSH9T+luQUfME1d31zZA9Pc4HFJK2RuWHxBnvVGDOWL6ZWvmhybeIyQRPlRbdEJb08RAslmlimCLxn9XTYBVpohJGq5gk8Ykb70OEGs7nL87t+ItUUuWz38BFOMk37n36PnGH7uT7EZfDUsbNsuW1ZzCUCe1pT3VWE4wJaOYoLzkpftFz8t1z35+5RowNdEz9M0qgLjuTpz2gczUIHMEeXYGrEGpq8vGNer7o8+uWkIxyiDUgPow3Bxf94nl23pIsprL2fxvQf6LzsCZjzEMru41RLqnEsSsj2Efe896g36Z41y4RPfogp7xfU0W+sDFTtX4j2PIsMTXEhYxuPfSsgZbP9FuA6yeFPDhJHZMFXEHMbzzctQYJgs7rdgt2OOImuFECSwajAuAUxkU1eQfnECEiS6zXwSEqo76ozePYnWrEN8SLLOSpPeO8kuiI5dTnON8BIqDGHtlxCB8rY2yJ66abxQHXinG3+KgqPCACHIeyFawkYYtUFbyTa6OU1W2RUcTb2RvewEJGpM9IffIKIlY+12kC9J7QggWFPSAKemg/Hur1bEuPbR2ijJjbT T0DMaVt7 nuWryyQpenLeQN+VCf6klURDaJUrOLF1cxHUgwJ55IYfN4/1jNaTl+rf0snjAPP4q8NWo59YDNQvZH3Qf442qVRmgX+LRC+YWvQMr35PrfS1UlUzjGlr02T1cp5xdkBwVigShKS0U8vD6KwhQ8P0S/PbixABP2qPju6+ELbiKgf91onjAy5Iz6KgJtXqjSvtwpv4RWu6+W2vUZpZFpi1RgPRljyYU+R/RTLVDNIh3vNE/o2zj6iu/KXsPv8LJqWxdkynTEZhBrzV97hcH/lJ9LagPiB3A6sx2YyP/Ck0M3Vqt7j1UM8Lja454uraYGlrvvXQX9EMdg2EYIfoDAW5Uf/Fww4w3v0plI7cTVr1FHOq32prro5ypFvDRApkdwxN+z5IBbn193zs7B49lNgQi1wsCBh6uHa3bT/LOcJQ+xyYyCx7hSSWyT2V9eJJn8YbGEYG9R7T8/RnLiq2FhONxiSVh3h6RZPqjjwaaA5n2EK96tJvI1LUFS3huG57IGmqUfcNMsU3m8knmTquMFz+AFGlNz9juNtB1KJKjbDeLRm0ovGNY/cDzNIck1Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_get_lockless() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 11 +---- arch/arm64/mm/contpte.c | 57 -------------------------- arch/riscv/include/asm/pgtable.h | 2 + include/linux/contpte.h | 1 + mm/contpte.c | 69 ++++++++++++++++++++++++++++++++ 5 files changed, 73 insertions(+), 67 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index e85b3a052a02..8a0603257436 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1384,7 +1384,6 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, * where it is possible and makes sense to do so. The PTE_CONT bit is considered * a private implementation detail of the public ptep API (see below). */ -extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, @@ -1430,16 +1429,8 @@ static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) extern pte_t ptep_get(pte_t *ptep); #define ptep_get ptep_get +extern pte_t ptep_get_lockless(pte_t *ptep); #define ptep_get_lockless ptep_get_lockless -static inline pte_t ptep_get_lockless(pte_t *ptep) -{ - pte_t pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(pte))) - return pte; - - return contpte_ptep_get_lockless(ptep); -} static inline void set_pte(pte_t *ptep, pte_t pte) { diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index e225e458856e..5e9e40145085 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -28,63 +28,6 @@ static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, } } -pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) -{ - /* - * The ptep_get_lockless() API requires us to read and return *orig_ptep - * so that it is self-consistent, without the PTL held, so we may be - * racing with other threads modifying the pte. Usually a READ_ONCE() - * would suffice, but for the contpte case, we also need to gather the - * access and dirty bits from across all ptes in the contiguous block, - * and we can't read all of those neighbouring ptes atomically, so any - * contiguous range may be unfolded/modified/refolded under our feet. - * Therefore we ensure we read a _consistent_ contpte range by checking - * that all ptes in the range are valid and have CONT_PTE set, that all - * pfns are contiguous and that all pgprots are the same (ignoring - * access/dirty). If we find a pte that is not consistent, then we must - * be racing with an update so start again. If the target pte does not - * have CONT_PTE set then that is considered consistent on its own - * because it is not part of a contpte range. - */ - - pgprot_t orig_prot; - unsigned long pfn; - pte_t orig_pte; - pgprot_t prot; - pte_t *ptep; - pte_t pte; - int i; - -retry: - orig_pte = __ptep_get(orig_ptep); - - if (!pte_valid_cont(orig_pte)) - return orig_pte; - - orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); - ptep = arch_contpte_align_down(orig_ptep); - pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); - - for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { - pte = __ptep_get(ptep); - prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); - - if (!pte_valid_cont(pte) || - pte_pfn(pte) != pfn || - pgprot_val(prot) != pgprot_val(orig_prot)) - goto retry; - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} -EXPORT_SYMBOL_GPL(contpte_ptep_get_lockless); - void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index ebfe6b16529e..62cad1b974f1 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -785,6 +785,8 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, extern pte_t ptep_get(pte_t *ptep); #define ptep_get ptep_get +extern pte_t ptep_get_lockless(pte_t *ptep); +#define ptep_get_lockless ptep_get_lockless extern void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #define set_ptes set_ptes diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 54d10204e9af..01da4bfc3af6 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -8,6 +8,7 @@ * a private implementation detail of the public ptep API (see below). */ pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); +pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); void contpte_try_fold(struct mm_struct *mm, unsigned long addr, diff --git a/mm/contpte.c b/mm/contpte.c index 566745d7842f..060e0bc1a2a3 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -42,6 +42,7 @@ * - huge_ptep_clear_flush() * - ptep_get() * - set_ptes() + * - ptep_get_lockless() */ pte_t huge_ptep_get(pte_t *ptep) @@ -589,4 +590,72 @@ __always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, contpte_set_ptes(mm, addr, ptep, pte, nr); } } + +pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) +{ + /* + * The ptep_get_lockless() API requires us to read and return *orig_ptep + * so that it is self-consistent, without the PTL held, so we may be + * racing with other threads modifying the pte. Usually a READ_ONCE() + * would suffice, but for the contpte case, we also need to gather the + * access and dirty bits from across all ptes in the contiguous block, + * and we can't read all of those neighbouring ptes atomically, so any + * contiguous range may be unfolded/modified/refolded under our feet. + * Therefore we ensure we read a _consistent_ contpte range by checking + * that all ptes in the range are valid and have CONT_PTE set, that all + * pfns are contiguous and that all pgprots are the same (ignoring + * access/dirty). If we find a pte that is not consistent, then we must + * be racing with an update so start again. If the target pte does not + * have CONT_PTE set then that is considered consistent on its own + * because it is not part of a contpte range. + */ + + pgprot_t orig_prot; + unsigned long pfn; + pte_t orig_pte; + pgprot_t prot; + pte_t *ptep; + pte_t pte; + int i, ncontig; + +retry: + orig_pte = __ptep_get(orig_ptep); + + if (!pte_valid_cont(orig_pte)) + return orig_pte; + + orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); + ptep = arch_contpte_align_down(orig_ptep); + ncontig = arch_contpte_get_num_contig(NULL, 0, ptep, 0, NULL); + pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); + + for (i = 0; i < ncontig; i++, ptep++, pfn++) { + pte = __ptep_get(ptep); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + + if (!pte_valid_cont(pte) || + pte_pfn(pte) != pfn || + pgprot_val(prot) != pgprot_val(orig_prot)) + goto retry; + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} +EXPORT_SYMBOL_GPL(contpte_ptep_get_lockless); + +__always_inline pte_t ptep_get_lockless(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(pte))) + return pte; + + return contpte_ptep_get_lockless(ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5E47C04FFE for ; Wed, 8 May 2024 19:24:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 491276B009C; Wed, 8 May 2024 15:24:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 440E26B009D; Wed, 8 May 2024 15:24:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2E1BB6B009E; Wed, 8 May 2024 15:24:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 10A896B009C for ; Wed, 8 May 2024 15:24:45 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 9B6E01C159A for ; Wed, 8 May 2024 19:24:44 +0000 (UTC) X-FDA: 82096205688.01.439C509 Received: from mail-wm1-f41.google.com (mail-wm1-f41.google.com [209.85.128.41]) by imf01.hostedemail.com (Postfix) with ESMTP id AB63840012 for ; Wed, 8 May 2024 19:24:42 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b="uLEtc8/o"; dmarc=none; spf=pass (imf01.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196282; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bID6WfA+atBgi3UdotMTiBHvNsJiSSNtZP3FlzKWl8Q=; b=CEAOCz/np4HBGk45TSv65Mm4o04FpS3ZIgA5zGOTRyfWFLaG3KF4Xu5NQYXkmQ+tzI5clZ wp3we1z8rbzAHHPDQBqRmBhN+fzvlX7B8LM2KkpefRhuWpfuol/Xx3nZQHff5Z0RHmbklP wtgZK8tJeoODwv9k3RscksTwRg22j2s= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b="uLEtc8/o"; dmarc=none; spf=pass (imf01.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.41 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196282; a=rsa-sha256; cv=none; b=nh19IDBcT38KBWMRrb0KPnehpXn+g86OAzcciW/8U5781n/M8wjpADYbTZ0dUFPK/Dg9bJ r3lTMLYZIjZDZoreeeoqv9wA3Xc/6xILmzObdv452Ihw2O3xYVDrI4mEUOyc6DOaYaovSK CcWxJxa3286QZf3BkWIlonB7Q7HqrPE= Received: by mail-wm1-f41.google.com with SMTP id 5b1f17b1804b1-41f9ce16ed8so1067025e9.0 for ; Wed, 08 May 2024 12:24:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196281; x=1715801081; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bID6WfA+atBgi3UdotMTiBHvNsJiSSNtZP3FlzKWl8Q=; b=uLEtc8/oMEzMe2XEqGj5TT6PBM+MtHk13ehyAmfmsRryCwBwFtnhqKEU801hfOhKVR /9LBiXhXYAqTc9J0ZZZuXYg7zxUQayHhHG6a+z1GIvnSzAYV7MHemeqomDcHaOTtSiDq NaA072OoT5/heaDf7k0P1nT71PeeTnf9GzlyMMoclFkct7VLBJ2cj+cgKigp0UwxPX8n rBXGy+bTIlhJJCV3CDPDmNRbGaKa19LE57Vm0i14gzg6cYYKkn2DhWVGN9gRWGjTDOnl EVq/6XMZPhPotBmKkVgEZhhStFLy/Bgze1rXyiCOanc43DDhZVG9LVpYxQKuKIRZox71 gdiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196281; x=1715801081; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bID6WfA+atBgi3UdotMTiBHvNsJiSSNtZP3FlzKWl8Q=; b=ePNvGb2bKp5ypS8lCH6N5Rh0m02oXy9hh8538HzA9rSNeu+zIj+q01bCLtkg1cRhDp mZNFmEDeZc1gCVcjoSaAzM+hY0Dc3YXJ2+T+GnpiT1/qeuqyZouJFPwGTZWD3VivLtHM dIegN5HZDZXMj1PJxLA2gupOhJbldykbfb9E3z3kqZqRLDPyKHCl2A2Emg43inqj76oq xls04P9zdW6svcmr2JhYmylaKb94c2KMKDdw4EaX0SNDAsuhmbR16Fri8TI1ajOYbJWH 2Xsha1H4BbC6q17L+Wp7TC7BXk9BDksJqjmh0/IjD2FQcgzT+VY8mblSJvEWFs1q8KXI Vj2Q== X-Forwarded-Encrypted: i=1; AJvYcCX6vI92icnxs4++6xyrewMGHV844AVMmCqINhffjq4VWAJnMXS0sPvkqU50gtrrVh+R4y0k8NI4EJf2DdB+mNpPnCc= X-Gm-Message-State: AOJu0YwQ6aLB1WWn0mPOEJP9oPef5Rlu+MZ8jJchOPMc8LLNkaGmk31Z k4sXXQ/yBla5RntFewoZFVv1+ZtiUad1qwiCJJrwCMyrrQUQNhSSe1IyBj8Y+lg= X-Google-Smtp-Source: AGHT+IFH7sC2GowLam3Jd+gaWz8gHH4sQsrwabW1J5xMuNqZ6dTlKzWFO1Hh7H4Mfqebgr3mIcqVow== X-Received: by 2002:a05:600c:1991:b0:41d:803c:b945 with SMTP id 5b1f17b1804b1-41f71309fafmr39869075e9.10.1715196281225; Wed, 08 May 2024 12:24:41 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-41f882089cbsm32567815e9.48.2024.05.08.12.24.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:24:40 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 05/12] mm, riscv, arm64: Use common set_pte() function Date: Wed, 8 May 2024 21:19:24 +0200 Message-Id: <20240508191931.46060-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: AB63840012 X-Stat-Signature: jc7d57rwqny4gn5xmyxq7pddte8jbfgt X-Rspam-User: X-HE-Tag: 1715196282-794489 X-HE-Meta: U2FsdGVkX187U+J8Aq03HCUNF/zvvX0JQ7t4eznsBK4oOLc60yReH05ba/gxn+9pwi4ZijE3Zw31p5uxfnZ6a92g1L8n54YVzYYaCr7AoTYepLFg6ZlcYr9P6AIAd8tYa/TbfXGPac1kw5dLNz5JPdtRfD/XQIsXBCUA5im1wN3coWlvMi8lfBcyLNfGXQfHMmfL7LCBz3atmM7r8hz/r+aM0P1QCyE+qF9cvmc0/UwNoHPRvFmZaIiCG2pdjt4g9eYxWGE6o4jxv0FU/PpKhoW3iBQJz0uBQZ41mFu9sIZ40ABsHeP0PRRotEj0ALB6hGEN4qsiSfLWZQKxvER9JIgPZ/YhffqhQNGEgqKK+yYA9QJAwunVa6gIOEj1lDxsfcHrKzGpCGNXc0grm0SEmVkooVGB3n6opMOsE5ZnQJ5q2jqdYFUCENufMGg4SyuxzxwvzAmTKsY3BQL1Al09cZBUaqI0mYRHSIijl4PVXSFeFknilrBAsORrbF4neV5enVaTNjllZVjYfbOX+/WZjy1lQnqj63S5uE5p2StebtdWLOHsgEAvhEvjtd+eYMGN8pKOv9LN636qXMe74f9I8Wl84hhWTxNqV+21LaG2fCNBrlkKbqoOLJuFmZ9A2FCaugxlnYRN8PtUnSTn23Qe9QXawDHgwchnYAzRTDWBwxwNwkngRwL/9Cy6/CaXX2Y2laUtF6tw5O/8Jzp1YZxa2tHjbrk1IjEF/ZI/LkVlcCsQV44fMvL1UXywTx8goY0PHkmRWv3DV5MJwDMKodPuGbxt3xrrMlZJ86UWI15st8iV1duMFiZ2YBN7MOny9KzSMMw0h/e3BMCj1k6b17nPbnrFID540LGMdgNP4G7SU36U4EuFURTCcOIrh0j8G+QIggPh0hZZIQS9uxozFdX1eW4b7B5G8v5FMrBqDo3MloID+JUSVGCYcyBxAyFh4JXTX2SAzL7SnzNUs1gcFtz WijD5LNU XJGtV5dUEx4VjE9Y7JueviGgHl5khtaVtfX+QycVfeO0lz5o4eD2uqtuxvPl7asm9DWs64mfp/mcgG80mrUw6o5bGPN1ys+rpqc8rO3ERYHomODYAi0Au4rmqnp5tKopruBuQ95eOlIIbMRFXalJt98MjbNVEAlAmxOuaTHS1T7Jl5xxy6ZAcYvLyzZk2d0L6aXTCYyM0Co3lmiNsHY8XFrSEycymi9jX5KjsjnYZspvo5okXLlOOriifpcaAu2bwJmtqdvaynWkqM34qPAUlusH5KvfTGbmQViyiBCFXXSuetRaCn201EjHZFFDvaHLICQUl28XVS5zIqcTkHZjsir6aDBvIsMqentAK5lTkmzrPwpWem7xa0WYBwobhcoYTunJgqDI0yfXTNi7sZrr9xgsM0l6h2+DM1DlbQizYxxmOnRTgz6lCmSATZyvAH2LxYGSSgGdUk0Bd04cmyQ3/JT4oylj1gbAzLVlBy22McUJJIozYUPuPQIt0PO3+C5T5zYs3/7/O4uapJQbhKVx877fUMkg7ekBwpd5QpritzvFDnZ5Nf8fiaDuJFXuEvRkbTJz9 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware set_pte() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 16 ++-------------- arch/riscv/include/asm/kfence.h | 4 ++-- arch/riscv/include/asm/pgtable.h | 7 +++++-- arch/riscv/kernel/efi.c | 2 +- arch/riscv/kernel/hibernate.c | 2 +- arch/riscv/kvm/mmu.c | 10 +++++----- arch/riscv/mm/init.c | 2 +- arch/riscv/mm/kasan_init.c | 14 +++++++------- arch/riscv/mm/pageattr.c | 4 ++-- mm/contpte.c | 18 ++++++++++++++++++ 10 files changed, 44 insertions(+), 35 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8a0603257436..bb6210fb72c8 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1432,20 +1432,8 @@ extern pte_t ptep_get(pte_t *ptep); extern pte_t ptep_get_lockless(pte_t *ptep); #define ptep_get_lockless ptep_get_lockless -static inline void set_pte(pte_t *ptep, pte_t pte) -{ - /* - * We don't have the mm or vaddr so cannot unfold contig entries (since - * it requires tlb maintenance). set_pte() is not used in core code, so - * this should never even be called. Regardless do our best to service - * any call and emit a warning if there is any attempt to set a pte on - * top of an existing contig range. - */ - pte_t orig_pte = __ptep_get(ptep); - - WARN_ON_ONCE(pte_valid_cont(orig_pte)); - __set_pte(ptep, pte_mknoncont(pte)); -} +extern void set_pte(pte_t *ptep, pte_t pte); +#define set_pte set_pte extern void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); diff --git a/arch/riscv/include/asm/kfence.h b/arch/riscv/include/asm/kfence.h index f303fef8591c..36e9f638abf6 100644 --- a/arch/riscv/include/asm/kfence.h +++ b/arch/riscv/include/asm/kfence.h @@ -18,9 +18,9 @@ static inline bool kfence_protect_page(unsigned long addr, bool protect) pte_t *pte = virt_to_kpte(addr); if (protect) - set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT)); + __set_pte(pte, __pte(pte_val(__ptep_get(pte)) & ~_PAGE_PRESENT)); else - set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT)); + __set_pte(pte, __pte(pte_val(__ptep_get(pte)) | _PAGE_PRESENT)); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 62cad1b974f1..4f8f673787e7 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -539,7 +539,7 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b) * a page table are directly modified. Thus, the following hook is * made available. */ -static inline void set_pte(pte_t *ptep, pte_t pteval) +static inline void __set_pte(pte_t *ptep, pte_t pteval) { WRITE_ONCE(*ptep, pteval); } @@ -551,7 +551,7 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) if (pte_present(pteval) && pte_exec(pteval)) flush_icache_pte(mm, pteval); - set_pte(ptep, pteval); + __set_pte(ptep, pteval); } #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT @@ -790,11 +790,14 @@ extern pte_t ptep_get_lockless(pte_t *ptep); extern void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pteval, unsigned int nr); #define set_ptes set_ptes +extern void set_pte(pte_t *ptep, pte_t pte); +#define set_pte set_pte #else /* CONFIG_THP_CONTPTE */ #define ptep_get __ptep_get #define set_ptes __set_ptes +#define set_pte __set_pte #endif /* CONFIG_THP_CONTPTE */ diff --git a/arch/riscv/kernel/efi.c b/arch/riscv/kernel/efi.c index 3d2a635c69ac..673eca7705ba 100644 --- a/arch/riscv/kernel/efi.c +++ b/arch/riscv/kernel/efi.c @@ -72,7 +72,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) val = pte_val(pte) & ~_PAGE_EXEC; pte = __pte(val); } - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } diff --git a/arch/riscv/kernel/hibernate.c b/arch/riscv/kernel/hibernate.c index 671b686c0158..97ed3df7a308 100644 --- a/arch/riscv/kernel/hibernate.c +++ b/arch/riscv/kernel/hibernate.c @@ -186,7 +186,7 @@ static int temp_pgtable_map_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long pte_t pte = READ_ONCE(*src_ptep); if (pte_present(pte)) - set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); + __set_pte(dst_ptep, __pte(pte_val(pte) | pgprot_val(prot))); } while (dst_ptep++, src_ptep++, start += PAGE_SIZE, start < end); return 0; diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 70c6cb3864d6..1ee6139d495f 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -155,7 +155,7 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, next_ptep = kvm_mmu_memory_cache_alloc(pcache); if (!next_ptep) return -ENOMEM; - set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), + __set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), __pgprot(_PAGE_TABLE))); } else { if (gstage_pte_leaf(ptep)) @@ -167,7 +167,7 @@ static int gstage_set_pte(struct kvm *kvm, u32 level, ptep = &next_ptep[gstage_pte_index(addr, current_level)]; } - set_pte(ptep, *new_pte); + __set_pte(ptep, *new_pte); if (gstage_pte_leaf(ptep)) gstage_remote_tlb_flush(kvm, current_level, addr); @@ -251,7 +251,7 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, return; if (op == GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); + __set_pte(ptep, __pte(0)); for (i = 0; i < PTRS_PER_PTE; i++) gstage_op_pte(kvm, addr + i * next_page_size, &next_ptep[i], next_ptep_level, op); @@ -259,9 +259,9 @@ static void gstage_op_pte(struct kvm *kvm, gpa_t addr, put_page(virt_to_page(next_ptep)); } else { if (op == GSTAGE_OP_CLEAR) - set_pte(ptep, __pte(0)); + __set_pte(ptep, __pte(0)); else if (op == GSTAGE_OP_WP) - set_pte(ptep, __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE)); + __set_pte(ptep, __pte(pte_val(__ptep_get(ptep)) & ~_PAGE_WRITE)); gstage_remote_tlb_flush(kvm, ptep_level, addr); } } diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index fe8e159394d8..bb5c6578204c 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -325,7 +325,7 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot) ptep = &fixmap_pte[pte_index(addr)]; if (pgprot_val(prot)) - set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); + __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); else pte_clear(&init_mm, addr, ptep); local_flush_tlb_page(addr); diff --git a/arch/riscv/mm/kasan_init.c b/arch/riscv/mm/kasan_init.c index 381d61f42ab8..b5061cb3ce4d 100644 --- a/arch/riscv/mm/kasan_init.c +++ b/arch/riscv/mm/kasan_init.c @@ -41,7 +41,7 @@ static void __init kasan_populate_pte(pmd_t *pmd, unsigned long vaddr, unsigned do { if (pte_none(__ptep_get(ptep))) { phys_addr = memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE); - set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL)); + __set_pte(ptep, pfn_pte(PFN_DOWN(phys_addr), PAGE_KERNEL)); memset(__va(phys_addr), KASAN_SHADOW_INIT, PAGE_SIZE); } } while (ptep++, vaddr += PAGE_SIZE, vaddr != end); @@ -327,8 +327,8 @@ asmlinkage void __init kasan_early_init(void) KASAN_SHADOW_END - (1UL << (64 - KASAN_SHADOW_SCALE_SHIFT))); for (i = 0; i < PTRS_PER_PTE; ++i) - set_pte(kasan_early_shadow_pte + i, - pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL)); + __set_pte(kasan_early_shadow_pte + i, + pfn_pte(virt_to_pfn(kasan_early_shadow_page), PAGE_KERNEL)); for (i = 0; i < PTRS_PER_PMD; ++i) set_pmd(kasan_early_shadow_pmd + i, @@ -523,10 +523,10 @@ void __init kasan_init(void) kasan_mem_to_shadow((const void *)MODULES_VADDR + SZ_2G)); for (i = 0; i < PTRS_PER_PTE; i++) - set_pte(&kasan_early_shadow_pte[i], - mk_pte(virt_to_page(kasan_early_shadow_page), - __pgprot(_PAGE_PRESENT | _PAGE_READ | - _PAGE_ACCESSED))); + __set_pte(&kasan_early_shadow_pte[i], + mk_pte(virt_to_page(kasan_early_shadow_page), + __pgprot(_PAGE_PRESENT | _PAGE_READ | + _PAGE_ACCESSED))); memset(kasan_early_shadow_page, KASAN_SHADOW_INIT, PAGE_SIZE); init_task.kasan_depth = 0; diff --git a/arch/riscv/mm/pageattr.c b/arch/riscv/mm/pageattr.c index 98c9dc4b983c..d623e4fc11fc 100644 --- a/arch/riscv/mm/pageattr.c +++ b/arch/riscv/mm/pageattr.c @@ -71,7 +71,7 @@ static int pageattr_pte_entry(pte_t *pte, unsigned long addr, pte_t val = __ptep_get(pte); val = __pte(set_pageattr_masks(pte_val(val), walk)); - set_pte(pte, val); + __set_pte(pte, val); return 0; } @@ -121,7 +121,7 @@ static int __split_linear_mapping_pmd(pud_t *pudp, ptep_new = (pte_t *)page_address(pte_page); for (i = 0; i < PTRS_PER_PTE; ++i, ++ptep_new) - set_pte(ptep_new, pfn_pte(pfn + i, prot)); + __set_pte(ptep_new, pfn_pte(pfn + i, prot)); smp_wmb(); diff --git a/mm/contpte.c b/mm/contpte.c index 060e0bc1a2a3..543ae5b5a863 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -17,6 +17,7 @@ * - __pte_clear() * - __ptep_set_access_flags() * - __ptep_set_wrprotect() + * - __set_pte() * - pte_cont() * - arch_contpte_get_num_contig() * - pte_valid_cont() @@ -43,6 +44,7 @@ * - ptep_get() * - set_ptes() * - ptep_get_lockless() + * - set_pte() */ pte_t huge_ptep_get(pte_t *ptep) @@ -658,4 +660,20 @@ __always_inline pte_t ptep_get_lockless(pte_t *ptep) return contpte_ptep_get_lockless(ptep); } + +void set_pte(pte_t *ptep, pte_t pte) +{ + /* + * We don't have the mm or vaddr so cannot unfold contig entries (since + * it requires tlb maintenance). set_pte() is not used in core code, so + * this should never even be called. Regardless do our best to service + * any call and emit a warning if there is any attempt to set a pte on + * top of an existing contig range. + */ + pte_t orig_pte = __ptep_get(ptep); + + WARN_ON_ONCE(pte_valid_cont(orig_pte)); + __set_pte(ptep, pte_mknoncont(pte)); +} + #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1491CC04FFE for ; Wed, 8 May 2024 19:25:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A0F896B009E; Wed, 8 May 2024 15:25:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C00A6B009F; Wed, 8 May 2024 15:25:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 887DB6B00A0; Wed, 8 May 2024 15:25:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6C4BF6B009E for ; Wed, 8 May 2024 15:25:46 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2A46516119E for ; Wed, 8 May 2024 19:25:46 +0000 (UTC) X-FDA: 82096208292.08.53DFF4E Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by imf24.hostedemail.com (Postfix) with ESMTP id 39F9418000E for ; Wed, 8 May 2024 19:25:43 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=KkmpE5rY; spf=pass (imf24.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.49 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196344; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nb289X82Pwy2psbtYSkuOYlUdcLMn+kmFJiTkQU5Z+c=; b=l9DMSJ/8Ofz4icx6FA96osGA+lO202WnZ+edwtGKvK5Okgya81ir8t4EIM7D99QG+TY851 uWP6ViO4714kSB1nnabHNqHPUWmIo8Ph1yjq1oV4K16tmXt1gs7LFNJ/WdeJlfU+TMphPL RY6tWkrJR44kDWqYOgqR5wGNzEJGLMY= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=KkmpE5rY; spf=pass (imf24.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.49 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196344; a=rsa-sha256; cv=none; b=xTyjofe9U1h7YrJTlMayXivWdzv9xuNaHhax88ts412iy27uKZg/NRUpQRXeekUlLh/t6g OhWq30prU9iw0lyYxcldPnuSk8Dtix0VhyuyJlYPh+MFj3HwZtpe+JzJtrFQ/TNhL4WuzV bqeB1+8XMWxeqnK4RAdLY1rv3N49tls= Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-34d96054375so36209f8f.2 for ; Wed, 08 May 2024 12:25:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196343; x=1715801143; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nb289X82Pwy2psbtYSkuOYlUdcLMn+kmFJiTkQU5Z+c=; b=KkmpE5rYHKhMjXD1Ywce+2xnh37X4KvVzZQSbJaI0MN9YoahrnNZSiuOvR9hdCR6YI 6pXvQz6X682lG7QAKddAsbv91wRD0y6X6bcdZZfNlZQNVA6MYUAHXVcQeWLU/UYEDKHg grzCHn4BeQbafL1Zb/q8BZD1Fo7+vXk5nqaXuY3dC14G9vpNyqvCR75PW5OUzjnpx/el 8v9tLG/lF4jn1SQgz3yF42ZisrnB2yWH8PjahkNwAYQObp995BYC4267qBGyk6mKXxYS risUidNRACNGaOyOqmHnHz1+7EihfJQONS3UCJtz+0GP2r7hWak0t5fS/CTQN7k67Z1Z FSaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196343; x=1715801143; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nb289X82Pwy2psbtYSkuOYlUdcLMn+kmFJiTkQU5Z+c=; b=Hg0Q/DB1N7Fh2n4IhSVYDGfP/rUU9Wzs/8jHqX/Ta55b5Ag2DYhTfAKyhQlQJKXZCs XtqxHBceXJUsJwoOaPbNa1ETLgyUpAl495aP2ZTx0NL54TS8e8qXs25TjyBp6F3RReC4 RPXlHshvuUrrRbcRQdp8tAFwPMz2pA6j9J7DL6uIMEQ8sqzGGHQKqwb/ASncYoXsm5al dYeiYU3W38UDCgTBLKSbIPvR2RWaqF6CsGBRfET2MWen+gBYH4c2YIuGpPEJwZKKGWxu cmKWDoCmSJaDAcKiOYa623arHP39JTYUqFfnWHdOrXO7OgsARDj79/+nHGWVt7ccslea 32bQ== X-Forwarded-Encrypted: i=1; AJvYcCWuSHHiefWPq8bZ8px6+ahDWuPchG2kUooGscKGYCvVDvcn8KCtHyIQAWQbazMCAlk87uadW+dEWu7MLwr1pH7Z2hU= X-Gm-Message-State: AOJu0Yw8r56wfSDnO2vDpv4EGTky+QA2CHTGKNrgw7LMbbab5Q9RI0j1 X0hcdTpJdW1tvG8Au/yhXXVZ4IpltR3aehigw8P/78Y6FGCy96qUis27pTRwjUg= X-Google-Smtp-Source: AGHT+IFUGL9dDH2gwgI9tPWACeaOFPW71lxlPF9m81I7+sLroC1dBTrL+wbzR1KIUMB3+0Tz7vFgZA== X-Received: by 2002:adf:a492:0:b0:343:a368:f792 with SMTP id ffacd0b85a97d-34fca621315mr2849241f8f.52.1715196342717; Wed, 08 May 2024 12:25:42 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id b12-20020a5d4d8c000000b0034e65b8b43fsm14038517wru.8.2024.05.08.12.25.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:25:42 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 06/12] mm, riscv, arm64: Use common pte_clear() function Date: Wed, 8 May 2024 21:19:25 +0200 Message-Id: <20240508191931.46060-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Stat-Signature: bhkg3c3unjt8zg99u34uujb4zesy7wfx X-Rspam-User: X-Rspamd-Queue-Id: 39F9418000E X-Rspamd-Server: rspam05 X-HE-Tag: 1715196343-594390 X-HE-Meta: U2FsdGVkX1+rXTWzyEYqha5oZZGI5sbFifqIPvpAwLbcU1dbnhfLjnkLBrtxkd0jgwG4VQmRsnuvnJ+Bm7r0rt+9L/2NBO8sq19/emwUCR8235XzEnNX7qfRkNMx0esJJgrdbKtb9EWzJCpzz/v46mz91bStM0uBApllb/7hxCVOdH4VNStMMwpHa3Yi8dTd1syrqbfz/f+jRbV3xtTO6rr79PTU5GyzCl1nlDgKa3reCKfGUUUNNapWQIGNWx4n3BKU91O1H+unl23rQHMfNZgXfxdT8U6RpX8wk+MCEO30YuOGBtnqD6i4dvrNwrA4yniHZjAWV/CP2CYCdy/ctzd/qnxbd+7kJ6ld6RdW9H7PAtwvt1F5470rksYonWwTPeHJyN2Hm+uqUsEshsgAN4g13iM2C8cUo0Kfkaw30NL7Zp4f8Qulni7wnblQxyGyI4IY0lWBiGsNJm3E8SB+sc1hRDq/8uXgAttpTGwaB9sCtAdA1JHjYPOGA2FQrLxvBFDRmvqRx2pP9QsYxbokzizMzrlLAkHWQ9NcAyNOu5qm009ISGgeesieWMOyDCu0k0V3mXRewO1UWG3M21lrDIGaUrk0Wg0ZKT+HsWsfBFSqP2KJbTM0dfgVBPaN08bV3R8LC0NAuQ4w44JTWNySCwhbWbuZiTXOIaOx/0A7ci69W2iuLIXAWwUXgURG4oOGDFdXJUpAR2yDFctJ5OK6LlWSGBc2BnguSGEU/TsPRQyDSQdQFwiUYQsgHm73d75V/hc4pAEiN9gHAhg5KB9+ucLbSuVC72rBTCQI7mqmsw2SXEuAr0KVYZhLz92pkxYzeyF9OeM4oOx8N6UCL+fkIV1L7GjPFrIeEHdlzovpSeHnnySgIGL8Gr69a5bSn2Om56H4L1fOsLU9aIkKK0gPlDcjMGqeCbEyD5j1MasGQ8wtOaMXJIGlVyOlwe5kMcv0q3nvVaLEtsVVEB6uPGa RG2JB3fO HzQI8WR3f4GOei6bGZazNHg10LQ6f/yfQ4SYkwagmhVuRaAqY/FJZRjEKjmY8VhF/GfGYPMekZ83Zo7/a8Ic7tlsGZtG2ywM+MqdDmc5GlkVq0b6cA1XTiutiUbnYShXNNu9o/GPe1vEr+v5Nq86nCMak4vX5NPtwJem2n4ADHmheF3KFap4q9Xm0j6qEQN04qLPkzhuS+/1iIEppwlJGP4JS/a+KmF0dyBp1UUyowCp47avy86aULv+/HI/a+p5Jojf7qYNVzbveSP81PIwUbbZUWGQd3VMCOfeiCFmhr4Y7ricC1Lm2wlVO5+WxXzW+7RlLREDCdP7jPJZ45kZ7basl+cUe2ceZG0o54LbfLg9zerI+LStsEEiEJ31QxJtIgEYT7d4b0mbr0HrVB4xNQOGss4YItbcxkATVAWwcaGV8bfuU1RdGGujwTywkTWT/8/5Pun4xMuP2jF381SAZ6sLMF23nvwNokrZqxbGFYxDTgfQgZ4tGW43d12mAt00fNcGHpWlK3I1a2CXogWjpnY/jnLszwkDqF8zYxjUkz30XdB1pMtq/VxvLmw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware pte_clear() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 9 +++------ arch/riscv/include/asm/pgtable.h | 4 +++- arch/riscv/mm/init.c | 2 +- mm/contpte.c | 6 ++++++ 4 files changed, 13 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index bb6210fb72c8..74e582f2884f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1439,12 +1439,9 @@ extern void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); #define set_ptes set_ptes -static inline void pte_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __pte_clear(mm, addr, ptep); -} +extern void pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); +#define pte_clear pte_clear #define clear_full_ptes clear_full_ptes static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 4f8f673787e7..41534f4b8a6d 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -792,18 +792,20 @@ extern void set_ptes(struct mm_struct *mm, unsigned long addr, #define set_ptes set_ptes extern void set_pte(pte_t *ptep, pte_t pte); #define set_pte set_pte +extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define pte_clear pte_clear #else /* CONFIG_THP_CONTPTE */ #define ptep_get __ptep_get #define set_ptes __set_ptes #define set_pte __set_pte +#define pte_clear __pte_clear #endif /* CONFIG_THP_CONTPTE */ #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear -#define pte_clear __pte_clear #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags #define __HAVE_ARCH_PTEP_SET_WRPROTECT diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index bb5c6578204c..c82f17b3060b 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -327,7 +327,7 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot) if (pgprot_val(prot)) __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot)); else - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); local_flush_tlb_page(addr); } diff --git a/mm/contpte.c b/mm/contpte.c index 543ae5b5a863..c9eff6426ca0 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -45,6 +45,7 @@ * - set_ptes() * - ptep_get_lockless() * - set_pte() + * - pte_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -676,4 +677,9 @@ void set_pte(pte_t *ptep, pte_t pte) __set_pte(ptep, pte_mknoncont(pte)); } +void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __pte_clear(mm, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31288C04FFE for ; Wed, 8 May 2024 19:26:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B707E6B00A2; Wed, 8 May 2024 15:26:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B20A96B00A3; Wed, 8 May 2024 15:26:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C2146B00A4; Wed, 8 May 2024 15:26:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 801AA6B00A2 for ; Wed, 8 May 2024 15:26:47 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2D8F612014E for ; Wed, 8 May 2024 19:26:47 +0000 (UTC) X-FDA: 82096210854.23.D7BA724 Received: from mail-wm1-f47.google.com (mail-wm1-f47.google.com [209.85.128.47]) by imf05.hostedemail.com (Postfix) with ESMTP id 32D4B10000C for ; Wed, 8 May 2024 19:26:45 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=HsqPgOFP; spf=pass (imf05.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196405; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7NXxlYw1RM9+WdFNtDaqsh0kbg9AU8PQDaN/vzezo7A=; b=0HhjkS7lbWHF4gDmYwotKG4DA1aDHPJ2LWc0yUR02D1h4pd+O0itp6Dke2b0VOyan+HgQD zZmeP8oL/LN09yIcldvY+8pWF1w3nJAg5MKvBRlnDXHTIsbz1RXdUnnLzfbbf5vdYxKyU/ K1cmdOSeEnHViz2BpL5MpK18lMUIBbQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=HsqPgOFP; spf=pass (imf05.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.47 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196405; a=rsa-sha256; cv=none; b=YyqcJdKMNT7YefqC1UY86UbHTCDRnduPgnKOH0eHeGJlv7WZ2TbYuFpXhcBi+u0N6ugPxC KfhBHXmi1mJSR/JSpv/c9EXCML91yIvsgnIgMnH/3ULmlgQ7yYRRM/oGyyxTIwXh2+YQsz bEKtjIQRU46O1ZFUhmCb6o82LmeVGyA= Received: by mail-wm1-f47.google.com with SMTP id 5b1f17b1804b1-41b5dd5af48so711345e9.0 for ; Wed, 08 May 2024 12:26:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196404; x=1715801204; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7NXxlYw1RM9+WdFNtDaqsh0kbg9AU8PQDaN/vzezo7A=; b=HsqPgOFPNEQuyHedLsnJS07Cn8zDRBI77vPsOxelulOwqtybJOhKSI4Z2C9ei8bO93 gkLNZoAFdClS7ird0DLxA3/IciET+ojj0eAfVhpMYEvL/MCfsKTJAwIrFtN1uhEXaHQh zRLhCr6pH0fjfd2UAPxG2R+7R34/M5NOTv/pX6NiXpzawako47/zuuc0cGFAHTUQtWA7 XNxQsvHtFitVGktY1HprrRf3hIu22A3l1iEv7aRza7M2Sg116fY/2tNPjj/fy3g/qKyz rbXDFVGX5gjbIR9dTkSHP2lyQ6MKmHH0GYgGSgaWVaP8afAZ0y1opM2HDNWjvxO7xmi2 ds9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196404; x=1715801204; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7NXxlYw1RM9+WdFNtDaqsh0kbg9AU8PQDaN/vzezo7A=; b=aDMTR/z/TNDKfYUgvIkmuJUUH3GoTiOCq6rPLlipU696lzgGN+7VR+JOnxGTSdNCQd x/+J6oADlna2QGSqpVVyAIsyu3OnUIWjJZLSeW4nfe/JBDH2DByNdBc36bNVy8dHRrLQ PSaK8LnWe+GBcRyqxTUc5iue1885EXdSAOlSggrgrpOpUPcLU/NNYYRCT3GzqzXFqEEb eDAlTuUMKwqvODCew8hARU0xdj7WrtIMyjU6EbbfD8GXV8KLQZkJSD2Vq7Vo7X1hHY/7 XS3xwKrNqlxw1u7Z25FpFKB1nYakY/ms6fUs4ordngTzoeSu+K7elci5Dk+Ab9iyU0Aa +J7A== X-Forwarded-Encrypted: i=1; AJvYcCV7W2Frm3J3VvSBzzNQPTNb/EgyfvMSu8A+gagi3s1dIJXbCRMJ18y9kxB7/tYkFjxEM6H3Y73UgNQYU3bEzquItuw= X-Gm-Message-State: AOJu0YyaZiWAp3tYuhZcv1nSOynpUtCyOjFgv3wGpKQyREbyGJAcWQGn D6EhVwPnPNtGMZbU9TleCOv8Rr+IDC3wMi4RacUevordU8hrUAvC8AG4zu4/dFw= X-Google-Smtp-Source: AGHT+IGACzzezThLxVMxwcYVQ6pqU/BhrbSjplAZdXD5UPpMsvfoJ1LwcNLet4jFhPFrbMRRJqyyWw== X-Received: by 2002:a5d:4522:0:b0:34c:bb79:452b with SMTP id ffacd0b85a97d-34fca62159dmr2733749f8f.52.1715196403826; Wed, 08 May 2024 12:26:43 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id o16-20020adfcf10000000b0034b1bd76d30sm15921429wrj.28.2024.05.08.12.26.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:26:43 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 07/12] mm, riscv, arm64: Use common ptep_get_and_clear() function Date: Wed, 8 May 2024 21:19:26 +0200 Message-Id: <20240508191931.46060-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 32D4B10000C X-Rspamd-Server: rspam06 X-Stat-Signature: u7ncsaunmmc858g6umg3ag5gd61s16eo X-HE-Tag: 1715196405-336581 X-HE-Meta: U2FsdGVkX19jmlvwOtH62M/HGl06Zwm6HQMOKWwYzRTXK0wm8t9imLJy412FlLWM+8bkqbMBrNF7FGTxTi8TThe8jAWwTCJ9OOkRHFi0dFrgu43l2CkmQno0JbNraJCjZbRhy7OBbfCN+M9TvnhtHUAQRPamEe3fHdkC9adjpTPWnNxVoovtJOUkCbprLn3UnJ6kNTOxehWs3a3732M17WtNHn+6fwtRqNyfzvYqiwEoETMp57p4vk9pY+QhkzDqiggCprymnLezXR3iDUHo3ICdYptJmmY0x6ihIeLLXkFHDu3lsmg0nhwy0mZcMFqK80yNmZNQqZtplRHQW8IXg6McK+PcqmFovF2ed2tbO1ONinouhut58E363Y6Yd8x2HtlLxQ1n0flFZwKBGuvmxSTsHcZP88jLdGNXqswdWpJIxyhuEO8Tqa6bJ9quoePHbK7adhVljY4VFPB0yndLBCrax+Q/x+enLmmjwVZ5iVd5O4LomF+f4Q8v7ipOWk0m2GgNQNoruV3yYBX95UMrLobWXfackvET3HCKn8xfQJrBJO+dL6DZMUf8wUAbDwSP8nVGLiw6TMjTv6xkJ3rkWANNLBoPmSs4YXCdlu3yasMXb91h2Oq13sfnTN1jYL0AtNh1CvbFu3KL32aE2jedlzlZExKHI7HNAMHV92cppxuLsBLagvOTsEkJ4XxlX+hyhMsnjVwPdIppQeEo17mRZbcEOWKJoZfzy9jDz1fYYV4aiLxHV3Tl/ydGjclT5akuRDQjmLw6qc5c3CA7R7X8BL2iKyaWTihJ9uaM9w1NQ9VcJdZb2mjF5/TwMClKmytoLZ6kxgOOw8YEgs41ekDoMUVzRnMoqEU6WKEp/IeidbtTOFHvgYPk5vtmqGJHcXZM8Hii5hW3ObYHMl9sX9GsBvaNW1HtHvSyDTriqWrHy0N1f02UIySvo5XhsCK4OaVGmq1tc2Yl8sWzeX0dV7r ipOiB+we 8WGaWARTgDXHJqDXfy/7eFn2FQtQGqUiqH7OH4qWYJuW4Mzy31nubIz/x7qMg99EZMejsh3B9CLWvCrvMiZmDhbf3dwMT+LQeXzjxYAFhZFMCqMA9puG9DTXORv+8lRq8EvgJEKjCxPwXmonCBRJ+UdWqWv06BkbdPpxRKLOesTGe1OAKH9xkyL+76uaXqygI/KFQeTPYsYMGkVeUPXKetY5KhoIp1nhfDPZ3wA9t3mPBs24xdH6wtDi8odpeh1MltGyHpb2USHHQk/oTg9jxYrmVRFAuCFJYyljsWuU812gL+G1NwLQN4nNVaiu5S6+MJ4Bp/PjpcudUS2Rch12BgJP4qUetvJ8tJrjkn91+QHT8pxc83PFR6gthaqMDUS/6k8SDA3ZBG6ucnQZsbOr0dPHN8voeN2CfBtSJaS23ZT/ReEhJc9lIN/Jn9KuwvQcAE+1odom6yz92gep9cOS6uRKvr1jizGHiNWD1W7B0TxawpQpAWXfD7Z9cAr7CYZvsXGtbR3CFcT1VxRYx1ZiD0x3GoKVlZjjRKwqZIMd6eMxft1o+uje2ua7aIA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_get_and_clear() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 8 ++------ arch/riscv/include/asm/pgtable.h | 7 +++++-- mm/contpte.c | 8 ++++++++ 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 74e582f2884f..ff7fe1d9cabe 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1473,12 +1473,8 @@ static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, } #define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - return __ptep_get_and_clear(mm, addr, ptep); -} +extern pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 41534f4b8a6d..03cd640137ed 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -794,6 +794,9 @@ extern void set_pte(pte_t *ptep, pte_t pte); #define set_pte set_pte extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define pte_clear pte_clear +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +extern pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); #else /* CONFIG_THP_CONTPTE */ @@ -801,11 +804,11 @@ extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define set_ptes __set_ptes #define set_pte __set_pte #define pte_clear __pte_clear +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +#define ptep_get_and_clear __ptep_get_and_clear #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -#define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags #define __HAVE_ARCH_PTEP_SET_WRPROTECT diff --git a/mm/contpte.c b/mm/contpte.c index c9eff6426ca0..5bf939639233 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -46,6 +46,7 @@ * - ptep_get_lockless() * - set_pte() * - pte_clear() + * - ptep_get_and_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -682,4 +683,11 @@ void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); __pte_clear(mm, addr, ptep); } + +pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + return __ptep_get_and_clear(mm, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 611B5C04FFE for ; Wed, 8 May 2024 19:27:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ECD1E6B00A4; Wed, 8 May 2024 15:27:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E7CD76B00A5; Wed, 8 May 2024 15:27:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D6BC46B00A6; Wed, 8 May 2024 15:27:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id BA1106B00A4 for ; Wed, 8 May 2024 15:27:48 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6E2F58121C for ; Wed, 8 May 2024 19:27:48 +0000 (UTC) X-FDA: 82096213416.20.BCF0F66 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by imf28.hostedemail.com (Postfix) with ESMTP id 90836C001D for ; Wed, 8 May 2024 19:27:46 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=CeSbsAPd; dmarc=none; spf=pass (imf28.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.48 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jcfQWqIQyCa6XQTZ9KciN3KqArzYv/m686IwSCSxNEA=; b=Q9KCUoSRQPWiBB80xDXoHoCOvFvtFcT2iPi8PHM6WhZqqpCN0hr4STkA0awvCIOvFAzx5S 6c1Cf+JPZZxbFGeXaMsOoRN0kNLMlVDOw0WnR3QYb++q/fvME6iD9E5ok4PnUuOB9dZkli axvi/N45+Dz/6OCnNqymWKfw1v8HyE4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=CeSbsAPd; dmarc=none; spf=pass (imf28.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.48 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196466; a=rsa-sha256; cv=none; b=Njf4Un40fhp7vqiRkkIiKsynzKf+1UyRuZAEsaS/rKXQaJCNLYSIIjXDPv+jWbW8O+zGrs rcsWuQfqGm4Lk1GfYEDkIsfgsyD3YXVij1NFiLs4oQfRCNxnYVNxYSfeQkwhhztNwTMhZ9 7P9EQzNbgNFFtirdvzcdarTPD/ZcFRI= Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-41ba1ba55ffso795205e9.1 for ; Wed, 08 May 2024 12:27:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196465; x=1715801265; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jcfQWqIQyCa6XQTZ9KciN3KqArzYv/m686IwSCSxNEA=; b=CeSbsAPd6jYQkzi8N2rE6EugEymvv6m+REtE1vv4a6rEl54ANAsowKyC+gwhJ+gRFn MxaVAdHOFTR3FulAvrQDMbFTuyXFmYYw6P3hgA2bUoUGxbA3d9bTca+qwio8y/uv00MY elAZbiFiIjBH4wWY+RjNPODS2XpvvVXrQlOjaZLpRxlMg2lBTIB8kVUkLb1+1h9G+lMI habTD1y8C9pAJ+3E/1xFDITlC/6c+K49ZbSO9Dc7BzdAumOu/EyH9LcLoIXZjJqOv87k fJ2W7oTMOeqzdbrNcmISIFop8qxaaGmFyQzmQhjGWzu9XlRvYWOHzOU1J2xKMHW6k4Wm 6n2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196465; x=1715801265; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jcfQWqIQyCa6XQTZ9KciN3KqArzYv/m686IwSCSxNEA=; b=sYymTI8/c/6XGjgW5J37k0cBV2kdEo/KCAahlU8odMbfkxaoRYepk9jmNUH2Vqe2Ox sHLS67LWSRSvKwAgnE8ZaQxiLu687YsrR1MHm575MjvpUvkwKl0HHqFzU8T3pNBfRX4c mRKCk3y+x+CmGig/pJjc2XZS7EI6zVeKtUQ+iCpALZll7j/vLABpzww6PESAfEPA/Hka SR0px8DQ8hD+v1yxF4cCId3gpDumjZrmDR5kR23zGLmfM4tk2F0YBiK3RZ9SxVRJ0Ebh 8MeMakxOpC3WM8hUoc7gjPMpGzNCS3Ng3dIiilemBD3ec3e0WguQPqs7aM5rMmC88Fh9 S+6w== X-Forwarded-Encrypted: i=1; AJvYcCVHfYBzk6xQ6pL7w/zo9AS4bISZ1OGtC3FK0qHwJAxttJrFyPK024jXKvANT0x9iGCfgJP30Cy2vI9HqLv4M23+uTw= X-Gm-Message-State: AOJu0YzKjdhoUfYOTq+6oAIFl4JTgHhpgky/Rw7k2cmJLA0b+zxC5Vxo 4mdKHpsjMIt851X6XBYsb11YQu0OMgy3qAi3H7OY4+zz+IvutGiPofZ7U59ttlA= X-Google-Smtp-Source: AGHT+IG713uLBQWbwM0MRsX1LOPSAYD3hwSeYc9+OS0/mfL1VayM0FBzmqDZCPlWciEj7G1Kaomx7g== X-Received: by 2002:a05:600c:3103:b0:41b:f43b:e263 with SMTP id 5b1f17b1804b1-41fbc12bdcbmr5274575e9.0.1715196465070; Wed, 08 May 2024 12:27:45 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-41f87c24f8fsm33175985e9.15.2024.05.08.12.27.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:27:44 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 08/12] mm, riscv, arm64: Use common ptep_test_and_clear_young() function Date: Wed, 8 May 2024 21:19:27 +0200 Message-Id: <20240508191931.46060-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Stat-Signature: rz8ffxp3q8qfya6swomjjheu1qkf6gez X-Rspam-User: X-Rspamd-Queue-Id: 90836C001D X-HE-Tag: 1715196466-59398 X-HE-Meta: U2FsdGVkX18qOiv1Wgec46uAA3AnhXw5kNMwQYq+sdfHSZYLVVitMWcJtWQ2vJi4AfjfX8uvY/ipdfHDUivxopzKfQUYzQZaPxcxBwgny4/lkmRi2N5xzX9uGuvG4SnFdDM9Hueuc2e6/gJhJ7sqAxrgWaVUsJ8SWKMBXfFMqu+qkD9QOruKDnHF1OeOT3rNvdM4PMwevs8ysYQfzBVKz/MR97IhNDcQTZR1LqToyJQW/4FJKYDIgL1GPJlbmAk/23WfUDb8i9koly0dU7FpgG2J2r+msuUJ/it43W8s4a1TUKlHbbkMSPMAHrrVBw7AM0Kh6yGniVEefPvKe61IcEyQ7Ozhxz1N6Z1rYXEwav0Os6bOeVneJEEA/GjHgYEb1h79OCnTHAWs9dDkPOoh38roNf8Gr/O/ZhlbK3ubYDghjO29dDGqGyOEjbCZRwNxyWieIs/zRbXS7SvZUU4i5AxT0L4uCR800tg+mn79u5tS5TRgx4OrO3v56VPbluAXltr7kl8XmrXki1pjv+ggGV/cRHrnvhZTaCV7uA/iXGT5rJv48hz1+7d0XKK6HGCXk1iNTbiLzS2o9CWDkT3yp/En5arZ+je5B36ih6DO9FsPF6FjQ6FIgj44XJpunRi+SYE+cds5NjSh4V2pfuWFpoUG77v039esuWH1gUMCOUNswpPG3pZ9dwkmn/ECNv+dVQPGsgZJvSJUxrcx0SRNbhtcAJ1eMj9GQRlmalcHnDptGJbOWGc6nfRhW6/k2FeI5PB60TVoXZd6RdG5euVWwdhKHmgD5cxmYrIfaFNcAuHL/LmYOU1S1o4giKTmmada2Bqs3cOc+LHt0QYjc7FWTr0u6H/l8hcAOSW+jM5Fv6+FpIgiI77K/fy0EJokLwGrrm1+Vq/JSuS9kwDv1WL2SHa3YwOpxDFl+oE0cjfvWvkQv2CUtVA0zBsmOQ9RIozfSA1YUOWTqy4O2jjRLKs wBblQojn zS6wgSCvBzWrh541qAP+YqunUlrvyyH2R8bnXk0Tq77EODx2RnonA8LDP1NqRJS3QRJmyDjjxsTfBFDfAmJiYoM/548azKh7f4iPY0I8kDG0UkstKDQArl/Q922fZM4Xb6JUaCQWQ0AlKA9VQzgPX02TvNQp+qb3r0hgGUDT/aPM31Gu0WZ7zmllO1HJbWDpP85UnpZy02OxfEy370qawkRw8xL7zrnypypoyYArdJ0wApjCBRBJWXB/oJAThGXHvJ6u13+lVfNG1TMwZF0ush0GGdbS/LGBBluPgTGfVqMFtgUwyrtZa6i3qUXWDBbcJEPvuxvZgjjeP3UJdgNO1hG3SKpWE6ZgyEyXJzlFnmb7e4KBhEwBTPlqXlpWIcOLUET9ewrDMw1cdgEWphcyWywVwMyri7w8pkyr/IL/Djb98rIO4dytf4he3etyeSPvGAo25ABF/2KnNMII2CbwdUf5ofuZmNaKSBV5VNLcuHWcHZU+dmkh59/PjDlqxuSY3epE1QAac+XrAV8RcfYrHNa5zU1RpSuwncHz1YUcww9sdi/eYAn1IWHOHTA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_test_and_clear_young() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 14 ++---------- arch/arm64/mm/contpte.c | 25 -------------------- arch/riscv/include/asm/pgtable.h | 12 ++++++---- arch/riscv/kvm/mmu.c | 2 +- arch/riscv/mm/pgtable.c | 2 +- include/linux/contpte.h | 2 ++ mm/contpte.c | 39 ++++++++++++++++++++++++++++++++ 7 files changed, 53 insertions(+), 43 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index ff7fe1d9cabe..9a8702d1ad00 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1389,8 +1389,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, @@ -1477,16 +1475,8 @@ extern pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - pte_t orig_pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_test_and_clear_young(vma, addr, ptep); - - return contpte_ptep_test_and_clear_young(vma, addr, ptep); -} +extern int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH static inline int ptep_clear_flush_young(struct vm_area_struct *vma, diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 5e9e40145085..9bf471633ca4 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -45,31 +45,6 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); -int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - /* - * ptep_clear_flush_young() technically requires us to clear the access - * flag for a _single_ pte. However, the core-mm code actually tracks - * access/dirty per folio, not per page. And since we only create a - * contig range when the range is covered by a single folio, we can get - * away with clearing young for the whole contig range here, so we avoid - * having to unfold. - */ - - int young = 0; - int i; - - ptep = arch_contpte_align_down(ptep); - addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) - young |= __ptep_test_and_clear_young(vma, addr, ptep); - - return young; -} -EXPORT_SYMBOL_GPL(contpte_ptep_test_and_clear_young); - int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 03cd640137ed..d39cb24c6c4a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -739,8 +739,7 @@ static inline void __pte_clear(struct mm_struct *mm, extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */ -extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, +extern int __ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, @@ -778,7 +777,7 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, * shouldn't really matter because there's no real memory * pressure for swapout to react to. ] */ - return ptep_test_and_clear_young(vma, address, ptep); + return __ptep_test_and_clear_young(vma, address, ptep); } #ifdef CONFIG_THP_CONTPTE @@ -797,6 +796,9 @@ extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_GET_AND_CLEAR extern pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +extern int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #else /* CONFIG_THP_CONTPTE */ @@ -806,6 +808,8 @@ extern pte_t ptep_get_and_clear(struct mm_struct *mm, #define pte_clear __pte_clear #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +#define ptep_test_and_clear_young __ptep_test_and_clear_young #endif /* CONFIG_THP_CONTPTE */ @@ -987,7 +991,7 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - return ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); + return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); } #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 1ee6139d495f..554926e33760 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -585,7 +585,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) &ptep, &ptep_level)) return false; - return ptep_test_and_clear_young(NULL, 0, ptep); + return __ptep_test_and_clear_young(NULL, 0, ptep); } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 5756bde9eb42..5f31d0594109 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -18,7 +18,7 @@ int __ptep_set_access_flags(struct vm_area_struct *vma, return true; } -int ptep_test_and_clear_young(struct vm_area_struct *vma, +int __ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 01da4bfc3af6..38092adbe0d4 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -19,5 +19,7 @@ void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); +int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 5bf939639233..220e9d81f401 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -47,6 +47,7 @@ * - set_pte() * - pte_clear() * - ptep_get_and_clear() + * - ptep_test_and_clear_young() */ pte_t huge_ptep_get(pte_t *ptep) @@ -690,4 +691,42 @@ pte_t ptep_get_and_clear(struct mm_struct *mm, contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); return __ptep_get_and_clear(mm, addr, ptep); } + +int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + /* + * ptep_clear_flush_young() technically requires us to clear the access + * flag for a _single_ pte. However, the core-mm code actually tracks + * access/dirty per folio, not per page. And since we only create a + * contig range when the range is covered by a single folio, we can get + * away with clearing young for the whole contig range here, so we avoid + * having to unfold. + */ + + size_t pgsize; + int young = 0; + int i, ncontig; + + ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); + addr = ALIGN_DOWN(addr, ncontig * pgsize); + + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + young |= __ptep_test_and_clear_young(vma, addr, ptep); + + return young; +} +EXPORT_SYMBOL_GPL(contpte_ptep_test_and_clear_young); + +__always_inline int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_test_and_clear_young(vma, addr, ptep); + + return contpte_ptep_test_and_clear_young(vma, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3993C04FFE for ; Wed, 8 May 2024 19:28:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F8C86B0096; Wed, 8 May 2024 15:28:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3816C6B00A3; Wed, 8 May 2024 15:28:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FBA76B00A6; Wed, 8 May 2024 15:28:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 019B46B0096 for ; Wed, 8 May 2024 15:28:49 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A8010A11B9 for ; Wed, 8 May 2024 19:28:49 +0000 (UTC) X-FDA: 82096215978.05.4BDFA95 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by imf15.hostedemail.com (Postfix) with ESMTP id B191CA0008 for ; Wed, 8 May 2024 19:28:47 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=DcwZ4N+D; spf=pass (imf15.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196527; a=rsa-sha256; cv=none; b=Wx82Ep5rmFizZAs6kLGbJbB0u9zwF5ht4jBWYzg6ADw0tVIRtlNHAm2kc9FZHJZbsX67BI ZRBWO0tSoL02rSl0Ipiivo4027pRe5iEiUv561ZmEj9B4uYOW9rdhfY0XL5mFfauwSjwnD ywbwVMuZQwZCwabR4GjMHTFaoErQXSM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=DcwZ4N+D; spf=pass (imf15.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196527; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=kNQMdOrqjnX1U7CNuAEsgjMUnFOfIYGFTGlQRPZ73Z9le3xd5QR2JzWOVeAduIZPjE9Rvw 2nN1bkpW8ZGEimuANBBtAvdTlboG/3Be7TAWmne1vOCZBb9cHwEVk6k9u+vfOckYnp4xfu X60jUZaNY0OFERbYwqUnXvoWG9s8mVU= Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-41ecffed96cso651965e9.1 for ; Wed, 08 May 2024 12:28:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196526; x=1715801326; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=DcwZ4N+Dcx7K0NT4ADocGnsK01grPBCMFsvFQzftONarcTngv6P6bc3v/UYFkGzOE4 kedZatmRfNLAYGANL6pzTO4b2TWYuGIL4Moy5B/QX5r0lFNt8Myppn6uO7cnI+o7glr0 D17lMV/I4jNuMK9flckaYZJz72r0saLY37qBzRDLr0WbWydF/QGXf4AGIR8uSiZ4SF3h /2EXyS4n/bmFN4PSuEwNoGFOeYtZ/V0VloZxhjVtt/2wLiEOe4NSrfQxTlUoLvJiNEOH a0E71P5zqE3YVLEzLBN/yrJJ/yaTvOayaDgmlFuKLKmbNBKhTVtQIh07HnFPVM9XDu7b EkfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196526; x=1715801326; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=wN+dwa6H7aVIOJ5DiMoQB//tGeUZCgHwpa3ZQpUamvI+1CGYQ/XZ1BizwOXr+ztqFg vZzjX/9MVT2gdPp67qWvGUMWsC+cnkbIN1eE8hOZneF4xpmreS8cJ7LuYsf2mXdN2wP9 8PCq2iAeJqzCQOhdq/YBiHC2DiKn9Mb3IdBwfObstS9PIbtbo90cUrDX5KJf9IWxihNa d8u7JXCObUCrR7wVOId1QNQfPIQG6zfAoL5KiwUAcml0E5Fs2FZrzP+wGMvjzv6TfnEK ERBdN8KLfDYXbYj4ATBVQeZlIilrIyk3li/HbcWEzdz5YRi01+69iRUOl82xV7ClNy3h JHeQ== X-Forwarded-Encrypted: i=1; AJvYcCXzCYrauhb4LEMUnYV1DlB0w2gGMM7/ENvx9DU+8xqmRZND0ihlZJjssQUWO41JvRc0VZB/L51MmUwDYYbFyifB7I8= X-Gm-Message-State: AOJu0YxBE+SJBlgMCB/NddCJsRt28p4sTf/sTwFYcNR/naKpCYP4vMV9 RpJRzo1UYzF6PXEfDWwOG7ggRK5GPxuTDuApUc/LAqUD3c5y41jmkt06yGCrxuE= X-Google-Smtp-Source: AGHT+IEyIjL1IsEMqXGLkJGiy2tkR0X5tVHC2TKSU1+3Y1r/Xzm4QA5qwQ487USNifryKvOLnO9vbQ== X-Received: by 2002:a5d:6350:0:b0:34c:65ba:5d43 with SMTP id ffacd0b85a97d-34fca621699mr2523239f8f.46.1715196526270; Wed, 08 May 2024 12:28:46 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id cx8-20020a056000092800b0034e01a80176sm16002694wrb.114.2024.05.08.12.28.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:28:45 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 09/12] mm, riscv, arm64: Use common ptep_clear_flush_young() function Date: Wed, 8 May 2024 21:19:28 +0200 Message-Id: <20240508191931.46060-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Stat-Signature: xfio9pwuc9s14ebnth84y6uiinffgs7s X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B191CA0008 X-HE-Tag: 1715196527-315464 X-HE-Meta: U2FsdGVkX18Q16S+s/bzQNEeWBeiVD4c70mD9ZBz4sTgy6swWN4b+0/osnD6uzT22DeEsUlfw6ZuR5KrcuA4ScH49NLx/J6x+aITMINTB+Qd6lbHySkLeCg3Pru72UgpuD09y2Hd3+AJNQ0JJJMhzCf/vjKt19wu8u1hP5vCAuA86uXNjscY7yGD4m2kX/GDlZiAZvX3MJBGh8AGaUzcmjhu6bsm0DMDx9WSlYtCOTZThgANH31QdAOuFPn2vQ4b6dzuPRfLyMexiOqg+vaq65aFsmT7lKlMZ0pjUy74rBzcScwMpS58a6f+3wZlPriIQI1/WxmswazmPsI2qA8bTfGPZD416N1ICQsqvdC5ni+PQfU1Imaz3yZz+Z0u7NafewAslbPNzNAXbQDpQfM9lY/V4AOgNGwQCL9kwJUuWXV0a05OLs7P5Fkrf08DUxb/mvnR2CJxZdqVD4/KFqTWq3Hr5TUGvM1QCkspd0/jaBax31MANllqiYxXJOpXdJRkGmzYhBZFpaqFAOAef5sX/671LgxYhsD6rjO/G3Lbz6yG2vIvWlF6RH7eJZgHrTXD0diiyVdwfR9c7w8XZALNow5gZf5gpyIMZzV3MJUjl4vWz7GV6x9FmomeLzGt9/wu4qH3vFo8Ffzl3IG70QDuEKXeSKjR/Fk7gzMLiT0kAlUssbj1Wecpjakn8BXXJtLKhY0cIwwxrsamNJ0zhKdU8J+jrF28/NExamNImbMUyjkz+JVP9hi6WFi/yUdXrAz+MAaiJFVJF0KyGl1v6gmnNEY15SLs6uvfHxLIMKQpzZxLCHwquRFHFw5nGEwnsHqXQ9M5GLQ0LtVWmBM4YOgPDsqyk0QoFQXgfCpYjNJnR/RbdCVk9H6IVrl90gegbDOJktKy5yFqIQUi5+ZCgq4cPXLwMTQ5Dk+UHj0a8QQONEOSZKzlxKOOZfO19a4vRq5EfnkEGvWbrc57Vhis9/1 KEKNKFf9 fXJ9qkbHMSScASPjSXiPAp7OXM1F4mwslYdeQkiEsEnfua5yfAET2dXECnJR3CG3YGQ9wdwqJeCfeHBgjyk4lI/n1pS5CBCHTQLK6k6UOwcpunPO7BolKPKr4w/A/C+wbnvllDueobg65CkYhybbTc35H3dv1w3UiTPmTNok7WXXvZFoWxsM6i+wV+RqLbJw7nQNDhaq4Emm5XY5Xphj3pg7yKGTtbxyHOICC6W4/hET9GPwnEXFPihejjaW/6fA1tGFPQMimfkuwlZbK2o5s9AuAnPK2qLaZymsQJMZ092nkADZb4luD6gqC1SbSelpv3eu0HrB8EHTDYC9dEvhTLwGROWwDxVcah8B8NuLmWpeizY2h5uxBzuuZLtb2VH8pyRiFIWQZARbmv0Iit98a5LUCE9mf62a7VUWk4aWp/cOusUDH9vBKKkSmJEcRLtvHoAm/6GNUFrh22pcnwSpRf9uWiEWhRPPHATh195Hz7KMGCuyhLslu4Z+R6UwPxPv/qrZSgDWwLP2T3MzMfLFOE/AJLdWP0PjhOr8UNI3jIyAr60Ao4tnkWh72og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_clear_flush_young() function from arm64. Note that riscv used to not flush the tlb after clearing the accessed bit, which it does now: this will be improved when we implement svinval support. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 22 ++++++++---------- arch/arm64/mm/contpte.c | 21 ----------------- arch/riscv/include/asm/pgtable.h | 12 +++++++--- include/linux/contpte.h | 2 ++ mm/contpte.c | 40 ++++++++++++++++++++++++++++++++ 5 files changed, 61 insertions(+), 36 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9a8702d1ad00..92c12fb85cb4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1389,8 +1389,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, @@ -1479,16 +1477,8 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - pte_t orig_pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_clear_flush_young(vma, addr, ptep); - - return contpte_ptep_clear_flush_young(vma, addr, ptep); -} +extern int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #define wrprotect_ptes wrprotect_ptes static __always_inline void wrprotect_ptes(struct mm_struct *mm, @@ -1616,6 +1606,14 @@ static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, __flush_tlb_range(vma, start, end, stride, true, 3); } +static inline void arch_contpte_flush_tlb_range_nosync(struct vm_area_struct *vma, + unsigned long start, + unsigned long end, + unsigned long stride) +{ + __flush_tlb_range_nosync(vma, start, end, stride, true, 3); +} + static inline int arch_contpte_get_first_ncontig(size_t *pgsize) { if (pgsize) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 9bf471633ca4..16940511943c 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -45,27 +45,6 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); -int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - int young; - - young = contpte_ptep_test_and_clear_young(vma, addr, ptep); - - if (young) { - /* - * See comment in __ptep_clear_flush_young(); same rationale for - * eliding the trailing DSB applies here. - */ - addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - __flush_tlb_range_nosync(vma, addr, addr + CONT_PTE_SIZE, - PAGE_SIZE, true, 3); - } - - return young; -} -EXPORT_SYMBOL_GPL(contpte_ptep_clear_flush_young); - void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index d39cb24c6c4a..42c7884b8d2e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -615,6 +615,8 @@ static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, flush_tlb_mm_range(vma->vm_mm, start, end, stride); } +#define arch_contpte_flush_tlb_range_nosync arch_contpte_flush_tlb_range + static inline int arch_contpte_get_first_ncontig(size_t *pgsize) { if (pgsize) @@ -758,9 +760,8 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } -#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep) +static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) { /* * This comment is borrowed from x86, but applies equally to RISC-V: @@ -799,6 +800,9 @@ extern pte_t ptep_get_and_clear(struct mm_struct *mm, #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +extern int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #else /* CONFIG_THP_CONTPTE */ @@ -810,6 +814,8 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG #define ptep_test_and_clear_young __ptep_test_and_clear_young +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +#define ptep_clear_flush_young __ptep_clear_flush_young #endif /* CONFIG_THP_CONTPTE */ diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 38092adbe0d4..76a49ac8b6f5 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -21,5 +21,7 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 220e9d81f401..600277b1196c 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -48,6 +48,7 @@ * - pte_clear() * - ptep_get_and_clear() * - ptep_test_and_clear_young() + * - ptep_clear_flush_young() */ pte_t huge_ptep_get(pte_t *ptep) @@ -729,4 +730,43 @@ __always_inline int ptep_test_and_clear_young(struct vm_area_struct *vma, return contpte_ptep_test_and_clear_young(vma, addr, ptep); } + +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + int young; + + young = contpte_ptep_test_and_clear_young(vma, addr, ptep); + + if (young) { + /* + * See comment in __ptep_clear_flush_young(); same rationale for + * eliding the trailing DSB applies here. + */ + size_t pgsize; + int ncontig; + + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, + 0, &pgsize); + + addr = ALIGN_DOWN(addr, ncontig * pgsize); + arch_contpte_flush_tlb_range_nosync(vma, addr, + addr + ncontig * pgsize, + pgsize); + } + + return young; +} +EXPORT_SYMBOL_GPL(contpte_ptep_clear_flush_young); + +__always_inline int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_clear_flush_young(vma, addr, ptep); + + return contpte_ptep_clear_flush_young(vma, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 090E6C04FFE for ; Wed, 8 May 2024 19:29:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 952F86B00A6; Wed, 8 May 2024 15:29:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DA156B00A7; Wed, 8 May 2024 15:29:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 753816B00A8; Wed, 8 May 2024 15:29:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 51BA46B00A6 for ; Wed, 8 May 2024 15:29:51 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CA93216079D for ; Wed, 8 May 2024 19:29:50 +0000 (UTC) X-FDA: 82096218540.09.F6B870B Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by imf03.hostedemail.com (Postfix) with ESMTP id E3DE62000C for ; Wed, 8 May 2024 19:29:48 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=One38r9T; dmarc=none; spf=pass (imf03.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=WF0jKPTkPY2gBN1mtEPQqXlfhfWLmVXpp5LvHmA+rO71vBlUShVH+cgCqT0G/K2RqW5dqb LUVOGUnoZc9JNGZukgicoTuNYNyFxi2LfZHG2bDjnpdCo0uvgRd/96tg9phQFZZKfPWh9x XcGEEqe5L5ilWjXhsOFr3zt5oVP0HRk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196589; a=rsa-sha256; cv=none; b=7SVL3xU3nMGsA35M8aaSScpBuaeHlMnBG3T/4mOexgEq++OUeQ9FlfXT0UURcJn+629Ao/ dbdAHaitJJUkHoDQuoPobCejuW8zTa/QZpB6+23X5J7RdaI5WPiPLrNnCRsYNd2QURiqQU Lqp9rwQpgHWnFxu27cA09vo/7+3L/v0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=One38r9T; dmarc=none; spf=pass (imf03.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-41ecffed96cso662195e9.1 for ; Wed, 08 May 2024 12:29:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196587; x=1715801387; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=One38r9TvfvDu8fsKgGJ+fqe7BSTLbYnO5Ng2o7I5Ptu7ayLxmbL8Fm53QVbvQTBOa GzdXEU+3RgYvtAbVORX0jVYQqxD1dme9qYgRzoGLIave2ZvF/mhNSZb8ypXQ0GkF0wqL B8LdTlB8MtFpHy/IT53KAJZKjtBjTLw4pvej4aaoRG9D+0jbf0kK5Cu+ZIRvTXmbC+Gn LuNtHGfJhXVAveObxNshDpo3OCHL7r0QSciFsFr/hLO5zoMQzG+mXEuHaEppBngpDljF BFHsZZ+F7zp/FRdq9wxD+D4kIvjlfqV0L8jQNET0KGIvP6A089VYSPjmsb1zHHVY2Ztq 5Yfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196587; x=1715801387; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=Mev99o9mHSe3U2RzM/zZ2qQxt1QTK+9JS7kYuSHX+CFLXPtPCeG424c63bXRGHBt+t AsEBvFb+VWtiWlT4/cNlPfAC1NPXgBXaHku40gKRRPY9FOhsylCRaCSGWHDYaGjFVuf+ gptMbNPiV9MtE/buPxlC3ddooeNZtcc73oxFdHpVWJP2Y5m90SxayIkXjHXsOBRKqhJd PYY3bLM63HoQxePhrRvVUvlVtCu8VgctrkIq1u4g/S36FwuKcMhSEjo9Q5ZAgS9FZGcf FbNc5VIA1TWvEAJBB1yvdypdWQmr0kZXkaI7w2TRnxDzAg+hoMe3jaP5q2O14dtShwTv nflw== X-Forwarded-Encrypted: i=1; AJvYcCUGchM6e81n9UsWDGES2UP66wi2u87shzaA88hjjwwKsNyPQmAZ5BKBqMd+bweoCT7qZGfXlIJY7dvdRhmBqbSfVuo= X-Gm-Message-State: AOJu0Yz5zbS+22SKvvpk0CxxFyNMqkq9k/zXCeIu2Yhb+y+Ll0po0Lyx MGaVon39ShP7bMlwiSp8DUfaBzvxhPTxrX9ptOQrWBHZGtK/ZEU70J0cPNp9ExA= X-Google-Smtp-Source: AGHT+IGQ61Vb4NuwkawA/0/DWqDCfOJMo7+D5f/tM30tLGXtIG6xkHE311QLb6FnYOpUeATe3M4jpQ== X-Received: by 2002:a05:600c:4e93:b0:41b:fc3a:f1ef with SMTP id 5b1f17b1804b1-41f71acca18mr25217385e9.33.1715196587426; Wed, 08 May 2024 12:29:47 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id d16-20020a05600c34d000b00419f572671dsm3314921wmq.20.2024.05.08.12.29.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:29:47 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 10/12] mm, riscv, arm64: Use common ptep_set_access_flags() function Date: Wed, 8 May 2024 21:19:29 +0200 Message-Id: <20240508191931.46060-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E3DE62000C X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: a4zqw1gijorn48ftrsq7j5b1u5whrryq X-HE-Tag: 1715196588-15193 X-HE-Meta: U2FsdGVkX18ypckEpArtPu7uREYyh2iditpl9ZDUbC4+K/9nzVpEZNX9OzP2daGFNRMG+95Nfq/CZf0EFg+NzrIWyn/MCIcenf4JPOZ7qcs6HNEcRyR9v0im44ccRoqSiDBTI7oSl5ZOYOiP42Awi7FIN5l/yex8U9FqEGfEPHbZrrUxpgxwG5168GIhKg54VqBWq9YcpvObzyY9ALkodxv8Vfr8bDRgCuAi83k/BwpSJGs7HvpxYpsVcjo0PPEf5x9eSnny+xTwqsCQi/h79k+WKfS5DlHQJiE4tBunkmSa60T+5MxXkzSUb5G6x/jJ04dbi0ig3umrde6LFVuwQaFW6EU3OuXOAN0FBaC8OJkKMfWADTfJCexE6eiSjHUIt4UiFmQMaVhoUPoOuqbbpLU8y7SMT7wv+mMTH9s6X/ELnLJS0JsgNLjBoB58DLZpAz9QXej427IGWq4zB6dg4hC02IrTVxMnREKRJjQQvGYwkA3GMn9LicWAAShBbNd2PwlS8foRe384LnnLoo698Hm/Y3oU5PJvSRopHuL/+J3Rv8bzsZjXj/vMpX/ykDJC3+yNfOlG57C0++IUZ9L1+R+Cv2Z4yC3Un3Ut8QQGrLffKzTKigW5MfwXfklP/0yGY75OghsXYl+rGSPe2Ee+t94C2OS3oYLtJ2rZ07CsDR1tUlqzbhTkCsQJJgdyPO/moVQsP/JG7QJ+YhIOtYzyQdApRQGNe42sN8DFJfiIiNJr4xpV1Zb50Cw2FAnCOVkPG83DO8ll1np4m+W1peTRXR6wkjeiZO3Kwg+Ob0QxYjqe1k53Xrm1hYn6dEhzP3VyryPWeriAEFTc1qIY5cNZwb16XodGv0xIBtWcyMe6tXQb95j0+iQwbKDoVoIt+iL1eZteQq/OdCxCcCr67xnTNs0dumal7md/xblSXgsF2yeEUjhR4yDR+luJnfCVzPahPMzcDbENNtqAfJ7RDcN ILb2QRGL BnSt7Om+mdOuC0A7DOqbO7aXS+9X3u5LzhwzxaxnAo1ZaEC6cK9pNkRShwOUTNtoWUi1RrPhtZcZ/UyJY38XN2g0+PhDHSgFvTUJcKgjvQmldMBRZUEqqBMEzJwAaqvzeCrLRGMGtTfF9uZUV1PYGGKdoC0sJl+rk1eXk1bnVE3RR2/+bwkqAN1y9HlJulUnhwB2fdMhHbpk+ElOEWf63n3W5DXehHiGOKq/c49ATxFwErmjyKvquX0sby1SHSWn19mqThbk2AHtrGVME+9THDL7jrOsaalsTcyrfdnxaeIeFBtON+nLSVU0fx5DlbavK2Y4GrWFLpT0jMmYvMZHsVkY645NDp+6aoELBVvNNGyDzOdmFev32bfTHXZBx8Awqt/qC0IgQPYUgfMfAmfjnM1bX+tCrb9Mb+t3OR3pvc3qx16Ju9e7/17BJ48mLpXQ8LB9GI0hMYasuLW6mcNK9VCsbLtnlWwdZJnedz5ttej0UpJHcr3zB120tC5xT36lcCeUiShlHqb+m0zDo22blI7VilO7yGGZxb3a5mjT9QAO5ekc1K8pcDos1Iw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_set_access_flags() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 19 ++-------- arch/arm64/mm/contpte.c | 46 ----------------------- arch/riscv/include/asm/pgtable.h | 10 +++-- include/linux/contpte.h | 3 ++ mm/contpte.c | 63 ++++++++++++++++++++++++++++++++ 5 files changed, 76 insertions(+), 65 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 92c12fb85cb4..6591aab11c67 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1391,9 +1391,6 @@ extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned int nr, int full); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); -extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty); #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) @@ -1512,19 +1509,9 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -static inline int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty) -{ - pte_t orig_pte = __ptep_get(ptep); - - entry = pte_mknoncont(entry); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); - - return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); -} +extern int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); #else /* CONFIG_THP_CONTPTE */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 16940511943c..5675a61452ac 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -62,49 +62,3 @@ void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, __wrprotect_ptes(mm, addr, ptep, nr); } EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); - -int contpte_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty) -{ - unsigned long start_addr; - pte_t orig_pte; - int i; - - /* - * Gather the access/dirty bits for the contiguous range. If nothing has - * changed, its a noop. - */ - orig_pte = pte_mknoncont(ptep_get(ptep)); - if (pte_val(orig_pte) == pte_val(entry)) - return 0; - - /* - * We can fix up access/dirty bits without having to unfold the contig - * range. But if the write bit is changing, we must unfold. - */ - if (pte_write(orig_pte) == pte_write(entry)) { - /* - * For HW access management, we technically only need to update - * the flag on a single pte in the range. But for SW access - * management, we need to update all the ptes to prevent extra - * faults. Avoid per-page tlb flush in __ptep_set_access_flags() - * and instead flush the whole range at the end. - */ - ptep = arch_contpte_align_down(ptep); - start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) - __ptep_set_access_flags(vma, addr, ptep, entry, 0); - - if (dirty) - __flush_tlb_range(vma, start_addr, addr, - PAGE_SIZE, true, 3); - } else { - __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); - __ptep_set_access_flags(vma, addr, ptep, entry, dirty); - } - - return 1; -} -EXPORT_SYMBOL_GPL(contpte_ptep_set_access_flags); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 42c7884b8d2e..b151a5aa4de8 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -803,6 +803,10 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH extern int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +extern int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty); #else /* CONFIG_THP_CONTPTE */ @@ -816,11 +820,11 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, #define ptep_test_and_clear_young __ptep_test_and_clear_young #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +#define ptep_set_access_flags __ptep_set_access_flags #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -#define ptep_set_access_flags __ptep_set_access_flags #define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect @@ -990,7 +994,7 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { - return ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); + return __ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); } #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 76a49ac8b6f5..76244b0c678a 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -23,5 +23,8 @@ int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 600277b1196c..9cbbff1f67ad 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -769,4 +769,67 @@ __always_inline int ptep_clear_flush_young(struct vm_area_struct *vma, return contpte_ptep_clear_flush_young(vma, addr, ptep); } + +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + unsigned long start_addr; + pte_t orig_pte; + int i; + + /* + * Gather the access/dirty bits for the contiguous range. If nothing has + * changed, its a noop. + */ + orig_pte = pte_mknoncont(ptep_get(ptep)); + if (pte_val(orig_pte) == pte_val(entry)) + return 0; + + /* + * We can fix up access/dirty bits without having to unfold the contig + * range. But if the write bit is changing, we must unfold. + */ + if (pte_write(orig_pte) == pte_write(entry)) { + /* + * For HW access management, we technically only need to update + * the flag on a single pte in the range. But for SW access + * management, we need to update all the ptes to prevent extra + * faults. Avoid per-page tlb flush in __ptep_set_access_flags() + * and instead flush the whole range at the end. + */ + size_t pgsize; + int ncontig; + + ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); + start_addr = addr = ALIGN_DOWN(addr, ncontig * pgsize); + + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + __ptep_set_access_flags(vma, addr, ptep, entry, 0); + + if (dirty) + arch_contpte_flush_tlb_range(vma, start_addr, addr, pgsize); + } else { + __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); + __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + } + + return 1; +} +EXPORT_SYMBOL_GPL(contpte_ptep_set_access_flags); + +__always_inline int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + pte_t orig_pte = __ptep_get(ptep); + + entry = pte_mknoncont(entry); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + + return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659111 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 971BFC04FFE for ; Wed, 8 May 2024 19:30:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2EA096B00A9; Wed, 8 May 2024 15:30:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2722A6B00AA; Wed, 8 May 2024 15:30:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0EC6D6B00AB; Wed, 8 May 2024 15:30:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id DE9956B00A9 for ; Wed, 8 May 2024 15:30:52 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A04E941117 for ; Wed, 8 May 2024 19:30:52 +0000 (UTC) X-FDA: 82096221144.24.7B03E29 Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) by imf14.hostedemail.com (Postfix) with ESMTP id 71ACB10001B for ; Wed, 8 May 2024 19:30:50 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=hJogDrsk; dmarc=none; spf=pass (imf14.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.179 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196650; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=SBbYYvkesewkw05bJXGNANkPaWIRy+AVHHrN8PBjPp3cSjve/qO2yr6t6tmFKfSSVicHKF t3hb6nHsnsi0cTR41uoa9Spmx98/M0h+Cefsk6f6xXMTSTs/1SX+V5nMt9StDwPuScJcYM w81S3y2s6C7XLOcF6EP2zzuwRXlZEQQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=hJogDrsk; dmarc=none; spf=pass (imf14.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.179 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196650; a=rsa-sha256; cv=none; b=vvPMMLHgjClrwusAGssQapsT6MXkRmsdF+wbMH+yV/+n55y/RWquUtylbAvzNii3dX3B7z RDDndcWxU/BAHHzNcL8VnM6Mqj775Tod4MznNHYkjYHQJacNFYJQ0IiB/AVmA96ugz0pFl 0iKod+1XBTSnzHeXOkmIuua9lQ66Eco= Received: by mail-lj1-f179.google.com with SMTP id 38308e7fff4ca-2e0a34b2899so1964781fa.3 for ; Wed, 08 May 2024 12:30:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196649; x=1715801449; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=hJogDrskHnX1/5z7ekXgMlOhQVLi7/DwlCBuvyTWoe9F/jxN4tAUjI6V1wN3OhJ0SZ v9qUNRDce4W/BlKAKVo8uQrQvNv5bXxOQtTMcxI6rXMZIKbj5zUSReNkHTcNPxtC1wqK a2VqoNUOeRyaDEUtCy//KmX1vym4ru4TiLKgmEyyXHckQIj3nWDemkA5EyGWI7Ai97/r 1J8LVBySiIhxaMxLQmotUwudh+cDalm9g31IW+lKYy7e49VpUc+taPrs7ZlwawnbK2Jk C4OG33ASKBa+HK2YOBWW+KLRH4+WQesJ3SEzgBp1yMh3Z/BsYl9rl3VBOnANB4kqu578 jODQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196649; x=1715801449; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K1s8quCXzhpwj2tdUrwB8sMaHFPnndxHFKfGkIDXtWo=; b=EILrzQT/t+wANSYH+eL5N6Qnnly5p9FkUjuOfXud7hpNxiICaTTGP4SFxJyXduSnsx laYwgrzdGKW8uCiWNiRgG4c9V5c7uz1jF3b8oQbMoyUkY71FS+NlWK8vGCZ6HWlewPlD D+QDZABmmc1J5BLuwSxcw2cI6sT1MU3SZUHjHFHteaqbieU/OuHkvd/NxDwBOnhCurwn EtR3GSHtpdoBrSeRITbB9SZvfhZH/P+U0h+gQfl1bpecwhDwMK9ygiPWOwY2xtwYkPub E1RHxYlazCPGDfcQeSAeQo2aMYX5mA/BCBXzXcnN7TzcgaeVNBkb/E9db+mR532dza0k 3ukg== X-Forwarded-Encrypted: i=1; AJvYcCUw+jC9RnVSfEO12imx1md2Ds4Hchk+TZa6CiqQWorhN6FosZiEGRzbe0BGD9DwoV646g4GTsqE7q+wh3H2RnHaaBc= X-Gm-Message-State: AOJu0YxS+MorfKuC56WOxmIVxNaSk4lnaUFyVeNqNCtTgtqCxO1aZUeZ cmKC/5br+x5SA3dAoa3dR5XMvKce4Xm+NCXan5BnaLXLm/8YPLuZHgDWWUcuApg= X-Google-Smtp-Source: AGHT+IGXrMpV3e8NWTuY3a54mvQHDm5XG0kBaIk0g0HXiaoXOXDbgZBz/5WZyQXp8WNGVe1a2PspPg== X-Received: by 2002:a05:651c:1541:b0:2df:e192:47ec with SMTP id 38308e7fff4ca-2e447081ef2mr37890971fa.29.1715196648606; Wed, 08 May 2024 12:30:48 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id b15-20020a05600c4e0f00b0041aa79f27a0sm3273819wmq.38.2024.05.08.12.30.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:30:48 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 11/12] mm, riscv, arm64: Use common ptep_set_wrprotect()/wrprotect_ptes() functions Date: Wed, 8 May 2024 21:19:30 +0200 Message-Id: <20240508191931.46060-12-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Stat-Signature: pyftaysm11q9qbzuege1zidtqykuptdg X-Rspam-User: X-Rspamd-Queue-Id: 71ACB10001B X-HE-Tag: 1715196650-501329 X-HE-Meta: U2FsdGVkX18+0H7SsV7kXRWzDGEMe0wUvRAfg0fbiARQMt0SYTCcExJhNVimkUNHzZWNPKY5E5d5bIBulw03KcVZRiFhADdhO+Wfs7z/NlvDkCsrMIdhFnAxHx4IhCmobX6pxOBL2e+XA0oa/9/DxWdSSSY/V5phsZxrfgHj5PRsXH2xJek3sjfOqZW2TwFNjuwK5t/dOEB6XJiIYCKox9sb4IBzJ7BDNgOdc0KBdheX7N//KIp3yqI+hnSp636OlgtduZ7Y24qrdvhA+6H42gXS51cDsTYOWuPpg0s9sv2ee4u3yUC20MsEDMPDls7doxRf1ryuQv1dwplszq25XIueSpTuYnQ+p1XzLcmATyJqxoj5sKIyUxtbk6bEVo0QoZHd1ygBIdq8I4c6jWkYyl2pUEyvbu1SiFF4GjLxT9tu/NSp3rmBOh+VxIqkNrg8uvS5VUvYKagsOrC6PNhfR3LOycC08mtLkKkuOZ9Sr16bewIfOLF90JapSdIhaSyQHkqVh8YLQk9mpdONtiZuXsSlqf7tgvIObRrJvD8sxA/qpR+f2BnOw3uHYnanFu3yFCRlkmYYZMByMvqjOkypb0p/mU8O9fu/GaXDFxYdV7rVnWXY9qmB7U6V2EplI6wPqcgmzBjQ0fTPL/2jXQj0ozDYHtSabK0xwePZ6zDcCTDR8VopMPFh7xIciXDZgjPN3FSqqENn1n803d/KJadJzqdnf3AyHyvDRAbfV9HlGYYCE6SCdlkHQirIkd00Q+ZpxYWAnTLTGSeR3Nb78C4sgMagFBpvxZsNNjlbOnyKxM+wBjoG4xc6afYBxhZo/BJnKEG6VrwBACrRZ9sbOyT4pQpHTc/0lG7brDN21DvqX3WBNXE3ze3aWD+QkW0eEmJKEb8yl8wKag4kQ5BzSx7kSA0a5FdnRcUvEcfi52k6bmlrd8/prRe+AbU5ejb6EbsaliDIIiFCx6lJhapn4gh 9S7g7sHX zQny/5q174E5fCgd2HsAADk1NkSVACUxzae8n25uer+UPL4dRrfMpDphi4TzSc1K9npZ4J6hDtSEjtOpfi+Yrpt4igQs6BfE/s8Ol6ck6b1NtuTQUwQmN8zlSBMcRdt4nBXtd9m1wv/Tdid9ZW+3+911j2udbuVwU6HW19D+Rt33maSRxvT7ZIxwN0624gKU2/TkZK/KPsMLK6hclwZdfbwvCeSem49Dr6GVbfpWryhgdCXsxI3lcYrbbSrlw5m8wIDN+sE9I0ndDFhV1yi3lq4pSrigclHVxfTtcFFLqxJvsWmHHG353t5zHoBM0aCYhmcaNY/k1Q5kT3mlGD2axRArJqJvVEAZvKKVDOVHiq/88xRAnZ12eP0Rq2HqDCMi6xjYLu6DexTaMeYLw365IzE9aDq9h05jRj9v7n9RN35vCDCudV2BFxQB4AU2D+cvMggoS42Ge4G7kRA6wRPJe5ioMSzYgmUdElocUiD8wA+0xS5yv4H7p/6kwFYgxVDrWdUcCLO52E5NRxbKcnFMR1X2CQiOKOIVopBAs9k36cAmQCTg9XR1HbeJo+g+mPxILy+ou X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_set_wrprotect()/wrprotect_ptes() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 56 ++++++------------------ arch/arm64/mm/contpte.c | 18 -------- arch/riscv/include/asm/pgtable.h | 25 +++++++++-- include/linux/contpte.h | 2 + mm/contpte.c | 75 +++++++++++++++++++++++++++++++- 5 files changed, 110 insertions(+), 66 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6591aab11c67..162efd9647dd 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1208,7 +1208,11 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline void ___ptep_set_wrprotect(struct mm_struct *mm, +/* + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware + * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. + */ +static inline void __ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep, pte_t pte) { @@ -1222,23 +1226,13 @@ static inline void ___ptep_set_wrprotect(struct mm_struct *mm, } while (pte_val(pte) != pte_val(old_pte)); } -/* - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware - * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. - */ -static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) -{ - ___ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); -} - static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, pte_t *ptep, unsigned int nr) { unsigned int i; for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) - __ptep_set_wrprotect(mm, address, ptep); + __ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE @@ -1246,7 +1240,7 @@ static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - __ptep_set_wrprotect(mm, address, (pte_t *)pmdp); + __ptep_set_wrprotect(mm, address, (pte_t *)pmdp, __ptep_get((pte_t *)pmdp)); } #define pmdp_establish pmdp_establish @@ -1389,8 +1383,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr); #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) @@ -1478,35 +1470,12 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); #define wrprotect_ptes wrprotect_ptes -static __always_inline void wrprotect_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, unsigned int nr) -{ - if (likely(nr == 1)) { - /* - * Optimization: wrprotect_ptes() can only be called for present - * ptes so we only need to check contig bit as condition for - * unfold, and we can remove the contig bit from the pte we read - * to avoid re-reading. This speeds up fork() which is sensitive - * for order-0 folios. Equivalent to contpte_try_unfold(). - */ - pte_t orig_pte = __ptep_get(ptep); - - if (unlikely(pte_cont(orig_pte))) { - __contpte_try_unfold(mm, addr, ptep, orig_pte); - orig_pte = pte_mknoncont(orig_pte); - } - ___ptep_set_wrprotect(mm, addr, ptep, orig_pte); - } else { - contpte_wrprotect_ptes(mm, addr, ptep, nr); - } -} +extern void wrprotect_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, unsigned int nr); #define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - wrprotect_ptes(mm, addr, ptep, 1); -} +extern void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS extern int ptep_set_access_flags(struct vm_area_struct *vma, @@ -1528,7 +1497,8 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_WRPROTECT -#define ptep_set_wrprotect __ptep_set_wrprotect +#define ptep_set_wrprotect(mm, addr, ptep) \ + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) #define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 5675a61452ac..1cef93b15d6e 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -44,21 +44,3 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); - -void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr) -{ - /* - * If wrprotecting an entire contig range, we can avoid unfolding. Just - * set wrprotect and wait for the later mmu_gather flush to invalidate - * the tlb. Until the flush, the page may or may not be wrprotected. - * After the flush, it is guaranteed wrprotected. If it's a partial - * range though, we must unfold, because we can't have a case where - * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this - * would cause it to continue to be unpredictable after the flush. - */ - - contpte_try_unfold_partial(mm, addr, ptep, nr); - __wrprotect_ptes(mm, addr, ptep, nr); -} -EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index b151a5aa4de8..728f31da5e6a 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -755,11 +755,21 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, } static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) + unsigned long address, pte_t *ptep, + pte_t pte) { atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } +static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, + pte_t *ptep, unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + __ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); +} + static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { @@ -807,6 +817,12 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +extern void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); +extern void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); +#define wrprotect_ptes wrprotect_ptes #else /* CONFIG_THP_CONTPTE */ @@ -822,12 +838,13 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +#define ptep_set_wrprotect(mm, addr, ptep) \ + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) +#define wrprotect_ptes __wrprotect_ptes #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -#define ptep_set_wrprotect __ptep_set_wrprotect - #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 76244b0c678a..d1439db1706c 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -26,5 +26,7 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 9cbbff1f67ad..fe36b6b1d20a 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -49,6 +49,8 @@ * - ptep_get_and_clear() * - ptep_test_and_clear_young() * - ptep_clear_flush_young() + * - wrprotect_ptes() + * - ptep_set_wrprotect() */ pte_t huge_ptep_get(pte_t *ptep) @@ -266,7 +268,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t pte; if (!pte_cont(__ptep_get(ptep))) { - __ptep_set_wrprotect(mm, addr, ptep); + __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)); return; } @@ -832,4 +834,75 @@ __always_inline int ptep_set_access_flags(struct vm_area_struct *vma, return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } + +static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * Unfold any partially covered contpte block at the beginning and end + * of the range. + */ + size_t pgsize; + int ncontig; + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + if (ptep != arch_contpte_align_down(ptep) || nr < ncontig) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + if (ptep + nr != arch_contpte_align_down(ptep + nr)) { + unsigned long last_addr = addr + pgsize * (nr - 1); + pte_t *last_ptep = ptep + nr - 1; + + contpte_try_unfold(mm, last_addr, last_ptep, + __ptep_get(last_ptep)); + } +} + +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * If wrprotecting an entire contig range, we can avoid unfolding. Just + * set wrprotect and wait for the later mmu_gather flush to invalidate + * the tlb. Until the flush, the page may or may not be wrprotected. + * After the flush, it is guaranteed wrprotected. If it's a partial + * range though, we must unfold, because we can't have a case where + * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this + * would cause it to continue to be unpredictable after the flush. + */ + + contpte_try_unfold_partial(mm, addr, ptep, nr); + __wrprotect_ptes(mm, addr, ptep, nr); +} +EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); + +__always_inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + if (likely(nr == 1)) { + /* + * Optimization: wrprotect_ptes() can only be called for present + * ptes so we only need to check contig bit as condition for + * unfold, and we can remove the contig bit from the pte we read + * to avoid re-reading. This speeds up fork() which is sensitive + * for order-0 folios. Equivalent to contpte_try_unfold(). + */ + pte_t orig_pte = __ptep_get(ptep); + + if (unlikely(pte_cont(orig_pte))) { + __contpte_try_unfold(mm, addr, ptep, orig_pte); + orig_pte = pte_mknoncont(orig_pte); + } + __ptep_set_wrprotect(mm, addr, ptep, orig_pte); + } else { + contpte_wrprotect_ptes(mm, addr, ptep, nr); + } +} + +__always_inline void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + wrprotect_ptes(mm, addr, ptep, 1); +} #endif /* CONFIG_THP_CONTPTE */ From patchwork Wed May 8 19:19:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31893C04FFE for ; Wed, 8 May 2024 19:31:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC3D56B007B; Wed, 8 May 2024 15:31:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B73E86B00AA; Wed, 8 May 2024 15:31:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A15BF6B00AB; Wed, 8 May 2024 15:31:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 836C36B007B for ; Wed, 8 May 2024 15:31:53 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3FCE581262 for ; Wed, 8 May 2024 19:31:53 +0000 (UTC) X-FDA: 82096223706.30.0F43D50 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by imf18.hostedemail.com (Postfix) with ESMTP id 57E011C0007 for ; Wed, 8 May 2024 19:31:51 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=lKgxP6Ym; dmarc=none; spf=pass (imf18.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196711; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3Mj4lZQ0sNzizek27x00kn2/PQ3ffbB6QFBBUZYNyzk=; b=q8E22F0KcV1/dSj1DEonFX86RCLPfu+pqc71ach0CfQUOMiBP7wsLxRBm6JYz6Zb8ZhSxP Syro4At/FxkchA4/TYQgaW+6mO7A8x2OsGjMY2Ec8sNQBxG9TjcIIxIZwGXoMu5NDzDJcS CfZq1CLjZbhexs1y4N8sAqLsNrowKMg= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=lKgxP6Ym; dmarc=none; spf=pass (imf18.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196711; a=rsa-sha256; cv=none; b=UgU6p55e/Z3O/LT8WXnp6EgJb3tEguy2pe8zXV/xtiaxNDYja3pLEsDFlPmAYDky034jrn 2RXFk7lx/D4gXrxnjWyO6wD0g53bhl7kehgHGAvaYkatmiHzVTSPX0T5W5HWryZMGeJQPC xTMX4cXEDDqqpK/zUsm2mUdu3TSlI1k= Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-34e0d8b737eso741277f8f.1 for ; Wed, 08 May 2024 12:31:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196710; x=1715801510; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3Mj4lZQ0sNzizek27x00kn2/PQ3ffbB6QFBBUZYNyzk=; b=lKgxP6YmDDYi1mGvohMkp/yQ8tRr7lPgLp3JrzPHCluw5Yb9MBqLRz99wlh3lX+UOK hKg3+HtQXhhiPzCO8f1+vTW7N8Hii77c4Pm7jYsMt0pWZrnwcHIG8klou0bIgEMjVwCY LG2RJ1zK4O1eXyFH1dO5l4MVB18ee7FuTY+mj0eTJz1N5zoZKbvsybBqi3TqpTeizVEv jaG5nXz0DRPBCPfRrUfOHn+XgdZ8ewZPrC5RfP7txaiyw29fabx/0oQmHFRir5RbAfGl nzez8D6rW+v+/Mov0+qxsZfMXVTRJ5Ff9KIBddC+wXACKcwA99R07GOursQXnvdppnZ/ QLLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196710; x=1715801510; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3Mj4lZQ0sNzizek27x00kn2/PQ3ffbB6QFBBUZYNyzk=; b=LVcaGx+3t8OP6p2Jxq5RwG7+yTcx6gO+skrg8zBbyH287JsTZNP/+eLbdwOZBx0mUy f3/EoIei6i3mCT7Gn/Q0Gl1xLe5QD7AjwCu/y705pT1Cz+nvgmD4mcCxu4tcbyxteKfu 18bldWpJbEwRvU3jl1K8jSF9MluuzkFgUhn/pr3IsRKQlquhHPY5HqBidfAJWnwEJgZe jlydQRlhuL8RBlLS9w+IRQ5E6T9HbP5Jinq9ctVjCXepT7xZIWnshz3RRCeHQr4xNIJ4 4ctbeVilTuDw4JVEXwd/db7P6QuT7rzMYdUcATsBOLxrejzAuzVU/aYyM2o0OFvC5n5a tiZA== X-Forwarded-Encrypted: i=1; AJvYcCXFj2vNUyOSGwv9HPKrRkkS4iS/AVfgLRYJMyRXgxZx0wlnFgSEiXu8/ZZiV/BKl+wGN3fXahkFohXUDSX6AiVBri4= X-Gm-Message-State: AOJu0YzH7jTS94Dt+Fjg4BJ7Z6bMey9wmGObjqqyllgs0C3kSigtBo50 Rnw3eY6BzNDkgIoOaWlVyalSKzZY80A81UhhWVF8VMCYWOQqGfWNXOhNdlnblTk= X-Google-Smtp-Source: AGHT+IGynnks5bQyXkefndTUc8PNt1+uZJK8w5+sVciCnroxfstzlHH5YDMjCv7Za3kPs8Z2dwdXzA== X-Received: by 2002:adf:fe8a:0:b0:34c:fd92:3359 with SMTP id ffacd0b85a97d-350185d57e7mr489056f8f.21.1715196709811; Wed, 08 May 2024 12:31:49 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id s2-20020a5d4242000000b003472489d26fsm15924162wrr.19.2024.05.08.12.31.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:31:49 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 12/12] mm, riscv, arm64: Use common get_and_clear_full_ptes()/clear_full_ptes() functions Date: Wed, 8 May 2024 21:19:31 +0200 Message-Id: <20240508191931.46060-13-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Stat-Signature: 88iknhdx1mwg7sn4mpoak3jy5h8ptyyh X-Rspam-User: X-Rspamd-Queue-Id: 57E011C0007 X-HE-Tag: 1715196711-634698 X-HE-Meta: U2FsdGVkX1+XU4oL6JpSzI8twtbHlk8756U8h/MZoOEOgeE177rzRAR2t5P26PdUHnfx2aHG+1eCaEluWG5n7i7PXHg94v+QS6h9BZQhvJZKsXb8XH0mLslCNXA2JovxlYkg7wX6NRew4OHUxdghTBW8RA9xkMXEDS7y0Q96P+N9So7tEPPTVuvuVVfaxb6SwFigubYQM5sLMTeCK1mV5rQ+X3qZSrmHyCadObJLfAYecMqOdoVBY2l+QXZEFBUw2H1308pXZk9BmXz65Y1Um2TG2ZzQw382ifuWvBpCfOKk9L0HRU7dwadbIiQPXUijQGVIsTzeHrGLy3hFyRYZgUMuXW2yygbQYmflrAhNKtZ5T2joCfORa98GqbB/2OERXuykosOG121L/ySUiSP4qdOj3fi5W3KvWdEx7BTZzY9waED1rOS9ESntgl5uvLljMMdX54G00Oljp7bQULD2fmD0G/wdrNuUImMlMFEZqgujQCPDj/w7kiMrTH31L+GQtp12oIbcv9SkKWZ/uwk6f4LUviMm8wZwXTwAxINbee0vBSf1cF2R/um3g3CRHeKiw1+Z3MoWyA0aCce+YkKjXEx+jn9TE8T1QAhqoVewlgyJMHj0kAo480zm31QCm1dtai/qN8lYXN9OgDvYObbaa/MEWHkoAFIkCeFLWdV1Ictcb59JAlX32v0PRMx9U4QUlT8QI4QLwp0rK4W4g0r5NxBSkWsAAvQJD0JAHCAxBijosgegT30R+/U0RDs0NFrLnhXQZ1IlfcVq9A5S7RE9V+pZfDoJi06dq22bT94umBLTlPc6cMfe3YdrVUZAO8/Ile5tUVyHRp2LbLsQdif88y3OroBMzU445/FQ3sB8VTJuCtrYCLlOABDE+CwY4EM/GWK9OLppfVgTO2mo9f7mnoDGV+rzu3wvZkBRKnjeh0BcB7kPRq7qWFnh+jP2Vu2P32PBTTAeJOR3DYHzslw rwAmuvyl JtDPFzgeBcTvyLq5hnLPyALnrvZerLjjn5yHU8nPjpqvk4ckjNdRbSc3rLqySkxB7uzDmO7MOYdWvtCq56WV2Yp4Eu4izEmXbIseM9flruhHV5Lk44esBfN/bbv0gULpmH1c3RvcEts9sXFEXeNRBY3nQm5gbNybeZmY/VHe0b1BAZCdY2kowPDA8ds9DHx0EeHvfsBPC7Zd1uj6OFHTTpxS0CusYkW9sybTvFbu5xcokNWP+6r0aVQ5ywkb+8Z+PZOPx5uPHDtsUI+MBjOajh7ZPfDrRIyFbG+unfmK2Nhb6RLEp/WPSDFIOWGBR/ZF2ie+V5kvmepOYn0evAmPGu4b/72caedbMZ7IifIamlYHbdB/OgZMipCy+BnJnOo3h6dOcR/4KFv4V84RSCmRpw7t+TkeCzettth+IPGBc7L5e7cypbKpF8ecynMhL04iL8k2LcZd9+ZfipOYWZOFTuWCO0B7SXVJRZY45LVfp3JmkKcLOn4JNSFhwVVEdQiR+dTlWkT2+XWGVkuzYIkr8iV3nk+/Kte0PhOpgHkIko5TxN/Rh88oeAhpHrjDMiassWJZg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware get_and_clear_full_ptes()/clear_full_ptes() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 41 ++++------------------------ arch/arm64/mm/Makefile | 1 - arch/arm64/mm/contpte.c | 46 ------------------------------- arch/riscv/include/asm/pgtable.h | 39 ++++++++++++++++++++++++++ include/linux/contpte.h | 5 ++++ mm/contpte.c | 47 ++++++++++++++++++++++++++++++++ 6 files changed, 96 insertions(+), 83 deletions(-) delete mode 100644 arch/arm64/mm/contpte.c diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 162efd9647dd..f8a3159f9df0 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1373,17 +1373,6 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, #ifdef CONFIG_THP_CONTPTE -/* - * The contpte APIs are used to transparently manage the contiguous bit in ptes - * where it is possible and makes sense to do so. The PTE_CONT bit is considered - * a private implementation detail of the public ptep API (see below). - */ -extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr, int full); -extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, - unsigned int nr, int full); - #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) { @@ -1428,34 +1417,14 @@ extern void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); #define pte_clear pte_clear +extern void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); #define clear_full_ptes clear_full_ptes -static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr, int full) -{ - if (likely(nr == 1)) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __clear_full_ptes(mm, addr, ptep, nr, full); - } else { - contpte_clear_full_ptes(mm, addr, ptep, nr, full); - } -} +extern pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); #define get_and_clear_full_ptes get_and_clear_full_ptes -static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, - unsigned int nr, int full) -{ - pte_t pte; - - if (likely(nr == 1)) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - pte = __get_and_clear_full_ptes(mm, addr, ptep, nr, full); - } else { - pte = contpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full); - } - - return pte; -} #define __HAVE_ARCH_PTEP_GET_AND_CLEAR extern pte_t ptep_get_and_clear(struct mm_struct *mm, diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 52a1b2082627..dbd1bc95967d 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,7 +3,6 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o fixmap.o -obj-$(CONFIG_THP_CONTPTE) += contpte.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c deleted file mode 100644 index 1cef93b15d6e..000000000000 --- a/arch/arm64/mm/contpte.c +++ /dev/null @@ -1,46 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * Copyright (C) 2023 ARM Ltd. - */ - -#include -#include -#include -#include - -static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr) -{ - /* - * Unfold any partially covered contpte block at the beginning and end - * of the range. - */ - - if (ptep != arch_contpte_align_down(ptep) || nr < CONT_PTES) - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - - if (ptep + nr != arch_contpte_align_down(ptep + nr)) { - unsigned long last_addr = addr + PAGE_SIZE * (nr - 1); - pte_t *last_ptep = ptep + nr - 1; - - contpte_try_unfold(mm, last_addr, last_ptep, - __ptep_get(last_ptep)); - } -} - -void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr, int full) -{ - contpte_try_unfold_partial(mm, addr, ptep, nr); - __clear_full_ptes(mm, addr, ptep, nr, full); -} -EXPORT_SYMBOL_GPL(contpte_clear_full_ptes); - -pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, - unsigned int nr, int full) -{ - contpte_try_unfold_partial(mm, addr, ptep, nr); - return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); -} -EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 728f31da5e6a..a4843bdfdb37 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -754,6 +754,37 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, return pte; } +static inline void __clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + for (;;) { + __ptep_get_and_clear(mm, addr, ptep); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} + +static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte = __ptep_get_and_clear(mm, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} + static inline void __ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep, pte_t pte) @@ -823,6 +854,13 @@ extern void ptep_set_wrprotect(struct mm_struct *mm, extern void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); #define wrprotect_ptes wrprotect_ptes +extern void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); +#define clear_full_ptes clear_full_ptes +extern pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); +#define get_and_clear_full_ptes get_and_clear_full_ptes #else /* CONFIG_THP_CONTPTE */ @@ -842,6 +880,7 @@ extern void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, #define ptep_set_wrprotect(mm, addr, ptep) \ __ptep_set_wrprotect(mm, addr, ptep, __ptep_get(ptep)) #define wrprotect_ptes __wrprotect_ptes +#define clear_full_ptes __clear_full_ptes #endif /* CONFIG_THP_CONTPTE */ diff --git a/include/linux/contpte.h b/include/linux/contpte.h index d1439db1706c..b24554ebca41 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -28,5 +28,10 @@ int contpte_ptep_set_access_flags(struct vm_area_struct *vma, pte_t entry, int dirty); void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); +void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); +pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index fe36b6b1d20a..677344e0e3c3 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -51,6 +51,8 @@ * - ptep_clear_flush_young() * - wrprotect_ptes() * - ptep_set_wrprotect() + * - clear_full_ptes() + * - get_and_clear_full_ptes() */ pte_t huge_ptep_get(pte_t *ptep) @@ -905,4 +907,49 @@ __always_inline void ptep_set_wrprotect(struct mm_struct *mm, { wrprotect_ptes(mm, addr, ptep, 1); } + +void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + __clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL_GPL(contpte_clear_full_ptes); + +pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); + +__always_inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __clear_full_ptes(mm, addr, ptep, nr, full); + } else { + contpte_clear_full_ptes(mm, addr, ptep, nr, full); + } +} + +__always_inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte; + + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + pte = __get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } else { + pte = contpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } + + return pte; +} + #endif /* CONFIG_THP_CONTPTE */