From patchwork Mon May 29 06:11:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13258138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57AB8C7EE23 for ; Mon, 29 May 2023 06:11:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5F6EC900003; Mon, 29 May 2023 02:11:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5A5D5900002; Mon, 29 May 2023 02:11:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 446CC900003; Mon, 29 May 2023 02:11:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 34DE5900002 for ; Mon, 29 May 2023 02:11:27 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F0EC9C01F1 for ; Mon, 29 May 2023 06:11:26 +0000 (UTC) X-FDA: 80842270572.11.E1C5A25 Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) by imf07.hostedemail.com (Postfix) with ESMTP id 3F7CD40015 for ; Mon, 29 May 2023 06:11:24 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=7B1W5MLw; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685340685; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=zliJFPKkBRA/n2GaYWcI8flBsJ+sn1iKXV6efEOvVuc=; b=EqeVvD3pa3+Jt7JwfYHMFAymBwdFwgRotGdKckf0Z4wGwXQRS8nDXdw9yYwfQgzfuaml6m 3ESyOs1hW4j511VHV/iqGWVm9ZZRpyHWhF5EA621DrJSLiR6J9+f2c9wjb8fx6RZNhKRF8 mRGHHN2fx7JTaJV1Aj27cOejGFogISE= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=7B1W5MLw; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685340685; a=rsa-sha256; cv=none; b=CPA11/ycWhRxMwqrX1nm7TB6oeHRETHqzgU6CcS3b1zZkf5EsJgFdxry45/tz8Tm9twt13 nBp7LI2ZgIKGTUiWU/Fp6Oc1fjGYyUS6BARRxiu67wmqVNzN6pVCLO+7C2xbtxMUG10QeO EojtPSn0lvbM1bNqTRB9AfdUIFmnwLc= Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-bad010e1e50so4546539276.1 for ; Sun, 28 May 2023 23:11:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685340684; x=1687932684; h=mime-version:message-id:subject:cc:to:from:date:from:to:cc:subject :date:message-id:reply-to; bh=zliJFPKkBRA/n2GaYWcI8flBsJ+sn1iKXV6efEOvVuc=; b=7B1W5MLwR/lNmx3BIWIEBSexKnAAtAeI5OObdohvQtJ0ya6QvwHlgYFcOox2PoWx79 AqTZ5kdBzWLNorOsK4QCVE297PN18hQKAQbCdVGnbvFaS7kyYm3WQLVA/nvW5TUXBwEK caAcy7rQi84u8SNtdxTLyllY4LVN73jaN501s4k727QQrqblWsE0z78z1nbHIMjpBe5J U7bsOrnIMfDvYrfJ38H4/OhgD5cA+Y2quHyXE8xkEJQ1/qi+xCB9+P6Rf/V3USkqhcRr ADhdVtWOZGV+AvVYyI9ulfCMHPe9IScFN/bL7d58cmN+M8/9WGgQpLLoAa3PpgIRJ6V+ lp1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685340684; x=1687932684; h=mime-version:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=zliJFPKkBRA/n2GaYWcI8flBsJ+sn1iKXV6efEOvVuc=; b=gJD6//j9rs583qxrQ/uriuIm5PFEJfxhoDjtSRl50Uc1rX75eZZ8n7IdN2xXgMJ45r pXDtOi5JYHCh9eN8JgSnqIx6YLqBpGVKu1+ONZ0bDUaHjSuRYpgQzNrjL7M0LqDsJAI6 v2rSAR8TBHysAwMMq7gkVU3ggv8xfGlEGSz/x0Oqc97NCnCBKA9DG9FcFKY0sJYbWqPT WwwEE8iwKzNahHkWml02OE6DfF96t15nPK+k5iVs7zl+1GJ3saBFCVt87hL5RLUuQnAQ zXPvTUoEXxtx2OjWAuRkThOjqe3GvDNO/5K/8Xjug/doNPpvodNqlPw66uiJsNDRWU5w me/Q== X-Gm-Message-State: AC+VfDwq8BTd8gK3/0xtJrnxeqVUXLGVhhKA6CcDRZ90rchDk3IceXEd H43rHvYn6UBsHw7d8w80nD1TPg== X-Google-Smtp-Source: ACHHUZ6vifKmHFoYdv3qrFkDElnMJbREaNq58bFiSjvre/dDskro6b5jD1pQPpN+OFNUHzxvXn3Qhw== X-Received: by 2002:a25:d4b:0:b0:ba8:6530:d561 with SMTP id 72-20020a250d4b000000b00ba86530d561mr10306665ybn.30.1685340684077; Sun, 28 May 2023 23:11:24 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id r63-20020a252b42000000b00b7b0aba5cccsm2703954ybr.22.2023.05.28.23.11.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 28 May 2023 23:11:23 -0700 (PDT) Date: Sun, 28 May 2023 23:11:07 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , Russell King , "David S. Miller" , Michael Ellerman , "Aneesh Kumar K.V" , Heiko Carstens , Christian Borntraeger , Claudio Imbrenda , Alexander Gordeev , Jann Horn , linux-arm-kernel@lists.infradead.org, sparclinux@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 00/12] mm: free retracted page table by RCU Message-ID: <35e983f5-7ed3-b310-d949-9ae8b130cdab@google.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: zynzsh9izfgexo7rifgaqhtz9tunmiyo X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3F7CD40015 X-HE-Tag: 1685340684-137589 X-HE-Meta: U2FsdGVkX18COeg3a7AEbDUjxDN6UxYgTzQmo+nSyWJrELD5zR2BUjsypodGC4mbhK8Y6q3QoViGjeAeiu4JtlzVm0QCOiIAz65MXJ2ZpFeor5ozTxxzQRXjstfc4Nr8T99UAOiOM0ymJHtQu52dZs2y19jOmupQiDix1tP1El6aa6Jp5wFqp6yAV7hTNmFVJj4Lin18HN5oZyuvrRBusyu+RqYd+P7mKs0MJPSWxCpRpvvprVo1zRC9nbt2MZzashV/oDjytnbYxqpKrwgS3luOHirHRfFuTEHh47YZJhjUfEFprGA/b1TyBylDTgKthZrwb2vbpGn5NOPqoRzaqFCnvKZ3LhEh/JtHFpk5u9yzqJ9Onq/N9v/6mWb+1NOppxBRW7Q5ebb5cC/FfxpeEH4UcyqSxT4tXsgnmhsLyhSMIDO43iPJW249Tw2Vq9qClJHf6bw0Yb6sFjCVaS3gQV5HGjqirYI8jSMOPb0gDqz9KNLpg/H/32hBupRcRQ3sDcz561tZu2EbtgZgYpZfdNuz2g4TQSAhi+VklWukSliU8v+brqRMopXVMldCkkBJedGrrCWh91Ris3HzkEQy8ng06BDwyUGgCmB+jeS5lRFallJDEGQM/YFX4TM/Ye3TDy+ZKBa9N1Qdax1ePt/9TDxa1AZQ+vNCCgdeDSjK9dnWcGSxDwsprD5yPJwKKQKwswxJ7QQXGxNu6A3D/Mvcu2X93RB2bIm8nslsBka4wVd0fzXD+/P+NXD9EMProZMWRGZD89n1iN7LoCifRXVIEt/7bPXjRBkCNOasaef2q/RcBH9Tp8aXPRHVeeAJ3Zxeytl0HZYIDx6ADgfrF1teUlXlcj2o4ELfYPgaaT5Br4CJENADWMQUfs+mJiyaerAWrL6FuVCB/VrZUXRVj9G+yevaN0MjvEC32dofne5StrLeH0lbgYwhweAG97Lmnq9EU+NK19KOemvOCfOFejy DLbpMTzr KPSs30L6Lo34zlFEVEP8ir9fAhcg4dvoDI+w4KxbDXYufB08bCbKJK0w250CwEbhDk8l5og2XdE/UPGbUZO8rre5ThXbsYRBPKW2AcZ+Uaryaf73R6dDqDJsUm+RIMpszIilVtPf/j2S+isuxUPZqUTGeBY9hRxTJwr3aKYdyMaw6hPlby93XKqmzWXxH4F2dJMkXdyjSbHiZ3O8oszJKAtSXX/H0GxIKTmjshsZLlhRekVIrzHASyA3ZxzuB58N3AcrMdisVYSko61wief1P46G3udaO/hHM/u10tpg3Uvk9pr2QmlfldsTOhpDujFl6wwAJ4XEKJlYzeyTxXclIFvEGcbpWH37UxSzb8epc5CFXCwj2yMJbF4iCHDhmsqU6VgQ9o57f0Yd0qgi3UWwfpCKeVcn0CqG6nf3uY0xob4DCVVfDZ2XNYuMrx1QyC456X5OG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Here is the third series of patches to mm (and a few architectures), based on v6.4-rc3 with the preceding two series applied: in which khugepaged takes advantage of pte_offset_map[_lock]() allowing for pmd transitions. This follows on from the "arch: allow pte_offset_map[_lock]() to fail" https://lore.kernel.org/linux-mm/77a5d8c-406b-7068-4f17-23b7ac53bc83@google.com/ series of 23 posted on 2023-05-09, and the "mm: allow pte_offset_map[_lock]() to fail" https://lore.kernel.org/linux-mm/68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com/ series of 31 posted on 2023-05-21. Those two series were "independent": neither depending for build or correctness on the other, but both series needed before this third one can safely make the effective changes. I'll send v2 of those two series in a couple of days, incorporating Acks and Revieweds and the minor fixes. What is it all about? Some mmap_lock avoidance i.e. latency reduction. Initially just for the case of collapsing shmem or file pages to THPs: the usefulness of MADV_COLLAPSE on shmem is being limited by that mmap_write_lock it currently requires. Likely to be relied upon later in other contexts e.g. freeing of empty page tables (but that's not work I'm doing). mmap_write_lock avoidance when collapsing to anon THPs? Perhaps, but again that's not work I've done: a quick attempt was not as easy as the shmem/file case. These changes (though of course not these exact patches) have been in Google's data centre kernel for three years now: we do rely upon them. Based on the preceding two series over v6.4-rc3, but good over v6.4-rc[1-4], current mm-everything or current linux-next. 01/12 mm/pgtable: add rcu_read_lock() and rcu_read_unlock()s 02/12 mm/pgtable: add PAE safety to __pte_offset_map() 03/12 arm: adjust_pte() use pte_offset_map_nolock() 04/12 powerpc: assert_pte_locked() use pte_offset_map_nolock() 05/12 powerpc: add pte_free_defer() for pgtables sharing page 06/12 sparc: add pte_free_defer() for pgtables sharing page 07/12 s390: add pte_free_defer(), with use of mmdrop_async() 08/12 mm/pgtable: add pte_free_defer() for pgtable as page 09/12 mm/khugepaged: retract_page_tables() without mmap or vma lock 10/12 mm/khugepaged: collapse_pte_mapped_thp() with mmap_read_lock() 11/12 mm/khugepaged: delete khugepaged_collapse_pte_mapped_thps() 12/12 mm: delete mmap_write_trylock() and vma_try_start_write() arch/arm/mm/fault-armv.c | 3 +- arch/powerpc/include/asm/pgalloc.h | 4 + arch/powerpc/mm/pgtable-frag.c | 18 ++ arch/powerpc/mm/pgtable.c | 16 +- arch/s390/include/asm/pgalloc.h | 4 + arch/s390/mm/pgalloc.c | 34 +++ arch/sparc/include/asm/pgalloc_64.h | 4 + arch/sparc/mm/init_64.c | 16 ++ include/linux/mm.h | 17 -- include/linux/mm_types.h | 2 +- include/linux/mmap_lock.h | 10 - include/linux/pgtable.h | 6 +- include/linux/sched/mm.h | 1 + kernel/fork.c | 2 +- mm/khugepaged.c | 425 ++++++++---------------------- mm/pgtable-generic.c | 44 +++- 16 files changed, 253 insertions(+), 353 deletions(-) Hugh