From patchwork Wed May 10 04:39:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13236317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FB4AC7EE22 for ; Wed, 10 May 2023 04:39:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23E336B0071; Wed, 10 May 2023 00:39:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C6876B0072; Wed, 10 May 2023 00:39:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 067B56B0074; Wed, 10 May 2023 00:39:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E4EFC6B0071 for ; Wed, 10 May 2023 00:39:28 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A876716021C for ; Wed, 10 May 2023 04:39:28 +0000 (UTC) X-FDA: 80773091616.03.07103B5 Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) by imf12.hostedemail.com (Postfix) with ESMTP id E4B0240005 for ; Wed, 10 May 2023 04:39:26 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cNF0lopR; spf=pass (imf12.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1683693567; a=rsa-sha256; cv=none; b=A5DJ27oXPX0dEFVMCohmz8wnXcu2JsXQYItS3AES3rU/s0m/N/T4YgsAq9cx1/qTd62Re7 ebrm7bRCpYZNL8SSoXJa8qB5vY/lQ2CwGumlnGqqggg8SggWUAxQTA7PfHJzN8aRVNv3C3 yFPrcEBQb2Hmu2UKBQlI+tg72pTKM+A= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cNF0lopR; spf=pass (imf12.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1683693567; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=pNtnMqwecxgAyJiNrU/av2u9SKsTTnkqGeLPsFfUGRY=; b=5QBxW5KtFEo5NMEi+cYM+cnkW18wxMM0CjZ0607Ge0eNvyiI+oQIoLYKrz2AYGgURDlQs3 MFJqhdGbtlFNIlQ/POb58AlAq17mQ3dXrKTXmYNKrQt1wXvKLX8cpcKprjiPPuNUHpN3T+ XPVF0HAZVhJNhTcPKgWLJEMZjUtFODI= Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-55a44a2637bso100390427b3.2 for ; Tue, 09 May 2023 21:39:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1683693566; x=1686285566; h=mime-version:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=pNtnMqwecxgAyJiNrU/av2u9SKsTTnkqGeLPsFfUGRY=; b=cNF0lopRgs5z+paNDd47dgtG8TBjwZyJSBqFjK6ksHmrX2HqmLMYrrKsDdWDE4galp I1RIa58CxJ4eWJPHfaXHtCDPLpNhfN/aCMBSDVyVUdoFoZEKW8H5uRAddSBQuOjqxl6G cijgVp5qOvAn/BuT5Kj3i7X1ZHy6xq4kMM1JJiMLi60Voi2WzpeuiP4d77oQFZQJJTpA DHkrQmjDtXwQxtNqUbLbFtn/koUU8sha9n5T/jPfXiKq7pKvhvnWuqWej7kSJ3OgTY5v znwXH+YUrN8s6GiaQCUoDOK5c9Mq29M3c3H1/3qfehC4bHIrJ+vibMAeHVK8iG01ICVy zeRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683693566; x=1686285566; h=mime-version:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=pNtnMqwecxgAyJiNrU/av2u9SKsTTnkqGeLPsFfUGRY=; b=ZGziWvPFlL1rEULNMYm0oM/HuUYjw3nszW/0KdhNKjdMhK2E97pElB/zofxq23a6z4 hMqZeKJEYRtmpGs9A5sOD02HN5ylCeL06ypZgUaRzF8BR6ajBdCOf3pgYKgOT829ppcd Ok915E3W+ATk2ww8v0fJvRA+rVfjouRdQ1b3DUQ+x+UAACyB9gbos/GttSvWc9QF2Q3C zgc7jNr65jXdElNLgDau9G4bWslc+OTwGpx3bnCpGgcArUAmoTP1Qz/LVd2poh1dqQa7 KhDTVG+5CGCyOeT+mbr3QWQVblCRYxFl+fsEPvkDb2Ybbx5PK/FXjbbqI90Fa/ifuSHm yvyw== X-Gm-Message-State: AC+VfDx99GQ7Pt2OjZCfg5mmsSTIivlx4Fv2apU/NSdKCYytt8KwDGxR 4d47SzwzRdgMpN50m5Fny92RCg== X-Google-Smtp-Source: ACHHUZ4tMEgG7AqlQIxMPNzAEIonGfK8+PPQUmJOyuPgdF6IMlPRJRM0QpGl7CYyO+0nNXPmO2L3jg== X-Received: by 2002:a0d:cc0b:0:b0:55a:9b56:acee with SMTP id o11-20020a0dcc0b000000b0055a9b56aceemr18605566ywd.38.1683693565918; Tue, 09 May 2023 21:39:25 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id i185-20020a816dc2000000b0055d5c626be5sm3775822ywc.26.2023.05.09.21.39.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 May 2023 21:39:25 -0700 (PDT) Date: Tue, 9 May 2023 21:39:13 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Russell King , Catalin Marinas , Will Deacon , Geert Uytterhoeven , Greg Ungerer , Michal Simek , Thomas Bogendoerfer , Helge Deller , John David Anglin , "Aneesh Kumar K.V" , Michael Ellerman , Alexandre Ghiti , Palmer Dabbelt , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , John Paul Adrian Glaubitz , "David S. Miller" , Chris Zankel , Max Filippov , x86@kernel.org, linux-arm-kernel@lists.infradead.org, linux-ia64@vger.kernel.org, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 00/23] arch: allow pte_offset_map[_lock]() to fail Message-ID: <77a5d8c-406b-7068-4f17-23b7ac53bc83@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: E4B0240005 X-Stat-Signature: 3b3t1ih8n8ztja5fjriu6n9t5twp3wp4 X-HE-Tag: 1683693566-504999 X-HE-Meta: U2FsdGVkX19OQJw4TGRbFYmJvB1r1rakNnaw1S1Wo9tqVi7+4g60Qs+Ulavby9a8UxBY2ZSPwsutn+oNS9q3aXl1PtVFAJ1s7NMDJJAlMlSHN9WOxfYzpGPh2mIsDpkJJXKdPsJYJ8QLi37wo4BBkfiKQqqx3xX4KxWKPB3S0BnhYJVNUzZQVY8D74MEbkDEJrPP4LbVAmdP+e7l+RL4khmKy+BeddH/O6XxdhuAsNxSF4oKalsqytJSfpyA7HjLJ0aqTJ8yV8xgBKe85RZijnFCCiamVv4uh6wO9amDFWB8GIzitI1Jr59wsM0xYz1Smz+aszc2NA3Gg5bCWM1LCah2fWcMMxMB7QLTxuSD/wGY22xKxvNuLhvxZIouUyLLXAMOFWhlKjB0HSF/y9lzH9Fs6anFV6IYDaD5bfXVPFaRURPKjs6W++16eEhKHnO5rFYO6p2gvevLVeukgjNw8d4yqBD9r2xPyMWNSN+HlDZB5aMxP/umyq5U0yr+YBnQlkR0D5DwsRS5Oi5HQxcbDAeg5Sp7WOCZ6aXWaGAdeQKRhqIhuNxLwfKJ3f5K8Xz2l8HYQbfOyh4lq59LEibkzpXRuJUDp2VtvJWwhsMIPIpdGwTiwhsBKLP6SotNuzfMkWO5kXmOAkWnzlm6zu5nXMGIr7Se407Ibk9rmTMQ7a9iuJrX92EjzxNxBNh8jeU2x85CEStunS1lLYDJa/1AmC/t4kN8Pgc/9IcNsozXXUyNEPHJ3SsZzakbLGCTdEcG12zp4T5ZTr/oHApWTx7OqJX1bTVRrMnw9B8vI909OSoflX5t/HN2/xwguJ16oD6NRwr6d1gslDV3j2dqjTgTHTaovB9NRy0QCkDFl3ZXNVmWSxin0dFUgEaSqi+D+ok+mUIWpBHV8hg4087us3BuEphyXJRYaTn24HHEno0NcmsRSQ7dVxi1io1YIJP/F4cd8Hvi7MFzufgwjfVpG/o fb83pxJe wvMpWhXQGWqk2tdtSopAwYk6WBpA9uFxESOeZe+drofYABoy9a7Sxa7Xs8f2is68xpTAvOmULrvPw9LQSNbh5P0hlw4mNJUaR2RutsLqOXbmphfdhlc/FTOI0VZauLmG3fGnUqzgcEKyfKG+ZgeIxXdAPQqArPszUeAr9KO9GaeD3sGYzzJiahGVjWk4CBrVccEJAGHHLXXuviMRd/SJeRZho3I14EX8JDrDUagj40ObtfKIvtTbMFURR/bj4OOAAfNhM0HqS8znYexevOyTv4tiqpU/sz52nQeuUfdLtEfA2f8EyILeWxAnRh2vcqgyCjObMk2PrNQBFLHq7+etYAzv91Hd6J4qP4yYrAP4hp/c5iUoGBxCnIO0Ac79nZSJU3ikqd4NqlDqNxHHFBMX3kJ0rKoxl6vkPiOxC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Here is a series of patches to various architectures, based on v6.4-rc1: preparing for changes expected to follow in mm, affecting pte_offset_map() and pte_offset_map_lock(). In a week or two, I intend to post a separate series, of equivalent preparations in mm. These two series are "independent": neither depends for build or correctness on the other, and the arch patches can be merged separately via arch trees (stragglers picked up by akpm?); but both series have to be in before a third series is added to make the effective changes (and that will add a just a little more in powerpc, s390 and sparc). What is it all about? Some mmap_lock avoidance i.e. latency reduction. Initially just for the case of collapsing shmem or file pages to THPs; but likely to be relied upon later in other contexts e.g. freeing of empty page tables (but that's not work I'm doing). mmap_write_lock avoidance when collapsing to anon THPs? Perhaps, but again that's not work I've done: a quick and easy attempt looked like it was going to shift the load from mmap rwsem to pmd spinlock - not an improvement. I would much prefer not to have to make these small but wide-ranging changes for such a niche case; but failed to find another way, and have heard that shmem MADV_COLLAPSE's usefulness is being limited by that mmap_write_lock it currently requires. These changes (though of course not these exact patches, and not all of these architectures!) have been in Google's data centre kernel for three years now: we do rely upon them. What are the per-arch changes about? Generally, two things. One: the current mmap locking may not be enough to guard against that tricky transition between pmd entry pointing to page table, and empty pmd entry, and pmd entry pointing to huge page: pte_offset_map() will have to validate the pmd entry for itself, returning NULL if no page table is there. What to do about that varies: often the nearby error handling indicates just to skip it; but in some cases a "goto again" looks appropriate (and if that risks an infinite loop, then there must have been an oops, or pfn 0 mistaken for page table, before). Deeper study of each site might show that 90% of them here in arch code could only fail if there's corruption e.g. a transition to THP would be surprising on an arch without HAVE_ARCH_TRANSPARENT_HUGEPAGE. But given the likely extension to freeing empty page tables, I have not limited this set of changes to THP; and it has been easier, and sets a better example, if each site is given appropriate handling. Two: pte_offset_map() will need to do an rcu_read_lock(), with the corresponding rcu_read_unlock() in pte_unmap(). But most architectures never supported CONFIG_HIGHPTE, so some don't always call pte_unmap() after pte_offset_map(), or have used userspace pte_offset_map() where pte_offset_kernel() is more correct. No problem in the current tree, but a problem once an rcu_read_unlock() will be needed to keep balance. A common special case of that comes in arch/*/mm/hugetlbpage.c, if the architecture supports hugetlb pages down at the lowest PTE level. huge_pte_alloc() uses pte_alloc_map(), but generic hugetlb code does no corresponding pte_unmap(); similarly for huge_pte_offset(). Thanks to Mike Kravetz and Andrew Morton, v6.4-rc1 already provides pte_alloc_huge() and pte_offset_huge() to help fix up those cases. 01/23 arm: allow pte_offset_map[_lock]() to fail 02/23 arm64: allow pte_offset_map() to fail 03/23 arm64/hugetlb: pte_alloc_huge() pte_offset_huge() 04/23 ia64/hugetlb: pte_alloc_huge() pte_offset_huge() 05/23 m68k: allow pte_offset_map[_lock]() to fail 06/23 microblaze: allow pte_offset_map() to fail 07/23 mips: update_mmu_cache() can replace __update_tlb() 08/23 parisc: add pte_unmap() to balance get_ptep() 09/23 parisc: unmap_uncached_pte() use pte_offset_kernel() 10/23 parisc/hugetlb: pte_alloc_huge() pte_offset_huge() 11/23 powerpc: kvmppc_unmap_free_pmd() pte_offset_kernel() 12/23 powerpc: allow pte_offset_map[_lock]() to fail 13/23 powerpc/hugetlb: pte_alloc_huge() 14/23 riscv/hugetlb: pte_alloc_huge() pte_offset_huge() 15/23 s390: allow pte_offset_map_lock() to fail 16/23 s390: gmap use pte_unmap_unlock() not spin_unlock() 17/23 sh/hugetlb: pte_alloc_huge() pte_offset_huge() 18/23 sparc/hugetlb: pte_alloc_huge() pte_offset_huge() 19/23 sparc: allow pte_offset_map() to fail 20/23 sparc: iounit and iommu use pte_offset_kernel() 21/23 x86: Allow get_locked_pte() to fail 22/23 x86: sme_populate_pgd() use pte_offset_kernel() 23/23 xtensa: add pte_unmap() to balance pte_offset_map() arch/arm/lib/uaccess_with_memcpy.c | 3 ++ arch/arm/mm/fault-armv.c | 5 ++- arch/arm/mm/fault.c | 3 ++ arch/arm64/mm/fault.c | 3 ++ arch/arm64/mm/hugetlbpage.c | 11 ++---- arch/ia64/mm/hugetlbpage.c | 4 +-- arch/m68k/include/asm/mmu_context.h | 6 ++-- arch/m68k/kernel/sys_m68k.c | 2 ++ arch/m68k/mm/mcfmmu.c | 52 +++++++++++---------------- arch/microblaze/kernel/signal.c | 5 +-- arch/mips/include/asm/pgtable.h | 15 ++------ arch/mips/mm/tlb-r3k.c | 5 +-- arch/mips/mm/tlb-r4k.c | 9 ++--- arch/parisc/kernel/cache.c | 26 +++++++++++--- arch/parisc/kernel/pci-dma.c | 2 +- arch/parisc/mm/hugetlbpage.c | 4 +-- arch/powerpc/kvm/book3s_64_mmu_radix.c | 2 +- arch/powerpc/mm/book3s64/hash_tlb.c | 4 +++ arch/powerpc/mm/book3s64/subpage_prot.c | 2 ++ arch/powerpc/mm/hugetlbpage.c | 2 +- arch/powerpc/xmon/xmon.c | 5 ++- arch/riscv/mm/hugetlbpage.c | 4 +-- arch/s390/kernel/uv.c | 2 ++ arch/s390/mm/gmap.c | 24 +++++++------ arch/s390/mm/pgtable.c | 12 +++++-- arch/sh/mm/hugetlbpage.c | 4 +-- arch/sparc/kernel/signal32.c | 2 ++ arch/sparc/mm/fault_64.c | 3 ++ arch/sparc/mm/hugetlbpage.c | 4 +-- arch/sparc/mm/io-unit.c | 2 +- arch/sparc/mm/iommu.c | 2 +- arch/sparc/mm/tlb.c | 2 ++ arch/x86/kernel/ldt.c | 6 ++-- arch/x86/mm/mem_encrypt_identity.c | 2 +- arch/xtensa/mm/tlb.c | 5 ++- 35 files changed, 140 insertions(+), 104 deletions(-) Hugh