From patchwork Wed Aug 30 09:50:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13370142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7920BC83F14 for ; Wed, 30 Aug 2023 09:50:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D17BD280044; Wed, 30 Aug 2023 05:50:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CA0B38E0009; Wed, 30 Aug 2023 05:50:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4328280044; Wed, 30 Aug 2023 05:50:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A12A28E0009 for ; Wed, 30 Aug 2023 05:50:31 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 800B7B286F for ; Wed, 30 Aug 2023 09:50:31 +0000 (UTC) X-FDA: 81180301062.18.653064E Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id 88BCF120022 for ; Wed, 30 Aug 2023 09:50:29 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1693389030; a=rsa-sha256; cv=none; b=2VMUXDCpQWdeICh6SnCBohI91tpElQSyVKVyhpk6cClCL52QNhZvHqjnH1o7tSjIhAzuvl 4ArkLlwFhBxXA0rHyxK2/VzQjLwlCFc9+PTJfJeLponjqIzZsCqaXdTZuzF4iW+mzh4SG8 16zuz5pBxaDC68zDKY4ztI4w/44OIdw= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1693389030; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=v9PgY6FMhffaDB3aS2bKoReBlh3WjS5OecNHlQ8Ww1I=; b=eKt4xc0VrQo+jA6Vs4WUX/YKk4whNavOgP05gb/WWleyywe7TZKt4bv1HApgzPyBo0IAdM hRzSfG0/BoI1IjCAqbMU/AFSGJok+wukmMQe5OBHvqK7CzavAoW8b/XYxbHUCqapIl6Fiq nxSJa1zwc0t3g9lJ3czNFld7f3Tc3io= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C95182F4; Wed, 30 Aug 2023 02:51:07 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C852A3F64C; Wed, 30 Aug 2023 02:50:25 -0700 (PDT) From: Ryan Roberts To: Will Deacon , "Aneesh Kumar K.V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , "Matthew Wilcox (Oracle)" , David Hildenbrand , Yu Zhao , "Kirill A. Shutemov" , Yin Fengwei , Yang Shi , "Huang, Ying" , Zi Yan Cc: Ryan Roberts , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 0/5] Optimize mmap_exit for large folios Date: Wed, 30 Aug 2023 10:50:06 +0100 Message-Id: <20230830095011.1228673-1-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 88BCF120022 X-Stat-Signature: 4m4fr13ng53gz7sio1ensk14ps1shesm X-HE-Tag: 1693389029-374832 X-HE-Meta: U2FsdGVkX1/HMQW/EemW4emdan3CNyie5uK9UpSS5EupIumViD2Wk2/5qKNw5WA45Op5o5Xk/ofv/sSSBXxIbSSDhhmmt4w8BA8YPCg7/Le7W7gG552RoWwVjVwXtvPEEW9sotg0XFTrJt9B7N1w24L0Cck+lpNGv31oygB7ddj4fNjF+9ED791PjB1/azB4Xus1qGJZgj6mkDm/ek03I5agVLvm2mToPR3q7lQAFLG3b6pfFdtpiJFMkABcZbwHYgqqumdLSfiKAbtwZJhFC2Ys6CksoAYgrzwFvhW3NtujoJoq2dpilpOeW23EkdW5ST/PxixltvkbrC/NIukkcir3wtxRdV2zlIel+68OHm46QDOW2ggI1tSkSjpIxRCUNsfTWDadcFFy7UtbC1LXhyXdDZ7cZPVUbZpHWYhcSPzKhcvlqqX1S+eRlKi/s0ZDFltBNmnFKXjrBbkDIYFTNpUerJe/0dblEsY4abZrpdSvY53OTVVkaV8/KPAOxPr/sHtJYMTan8MaRscA47ntRfhFCVeVUD7L4emlPxJxs6CsYNYidOrNNYGtDLAfxbwQMClJ9UjTitpej/EplcndZz2fni8Ur1fre5IJ1vUQaCBdzL1P/tAwRwNSJ9rKrzujQ6CG+MjwrS+CbT15BPKbNuFuiirjOya5qCh3ggwa0ULaN7hiqa3tjGKCJVuCDKWjuc0HhtV7Jslu56k+iro4zOK4iW8l8b2c3Qy/QhZvNvlz85Gnj2FxeQcNOMy5DdNmNYOmx8c50VD9y1oSuqWQu233LOk2qzjxxTki7+6cANd+4c+6sXdqlhFULXQgkgpk8wLQ5eB1YhBu9m8X7YjqCdm+FCKzjM/rlSU/iTs0C5toxBpyz8hNPW2TCqyroSZ5lZIQdIiS1OKtcD7kbbIk15+ZEjOH8tLJ0TFwqP0a0w7YCw2MchZ5zPud6Cc/FXOWn81BID/L7cpI46y8eC9 YtdkQyK0 A5RiDVAnpILgrMkHVnhN4Aj6R0m8TOhULjnbsj77un2Oru/tqRCn91Y98KreJheUcvNcFYjUl6oofE8PGS7sg/ZCSJYMdxRxfcqbo32BRltLMuZJh72608tk3nbWY7aEBISmQPjFDMAs0XfxOMg38Es3jzlA17Fao7moBDtqVLTazppnJ9GufnI4mQjgX4b+8uPK+xNaT7JO4pfdS95/6rNTJwlK/S2DDeORZqqfLsPeaMuxKfjbp9AxKRPkMgYBS1c99633166HqzPnPTC2SWiZQDkziepOpeuvuZr5EO2wG/LkhCiTeTarj21lh24w/LiI0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi All, This is v2 of a series to improve performance of process teardown, taking advantage of the fact that large folios are increasingly regularly pte-mapped in user space; supporting filesystems already use large folios for pagecache memory, and large folios for anonymous memory are (hopefully) on the horizon. See last patch for performance numbers, including measurements that show this approach doesn't regress (and actually improves a little bit) when all folios are small. The basic approach is to accumulate contiguous ranges of pages in the mmu_gather structure (instead of storing each individual page pointer), then take advantage of this internal format to efficiently batch rmap removal, swapcache removal and page release - see the commit messages for more details. This series replaces the previous approach I took at [2], which was much smaller in scope, only attempting to batch rmap removal for anon pages. Feedback was that I should do something more general that would also batch-remove pagecache pages from the rmap. But while designing that, I found it was also possible to improve swapcache removal and page release. Hopefully I haven't gone too far the other way now! Note that patch 1 is unchanged from that originl series. Note that this series will conflict with Matthew's series at [3]. I figure we both race to mm-unstable and the loser has to do the conflict resolution? This series is based on mm-unstable (b93868dbf9bc). Changes since v1 [1] -------------------- - Now using pfns for start and end of page ranges within a folio. `struct page`s may not be contiguous on some setups so using pointers breaks these systems. (Thanks to Zi Yan). - Fixed zone_device folio reference putting. (Thanks to Matthew and David). - Refactored release_pages() and folios_put_refs() so that they now share a common implementation. [1] https://lore.kernel.org/linux-mm/20230810103332.3062143-1-ryan.roberts@arm.com/ [2] https://lore.kernel.org/linux-mm/20230727141837.3386072-1-ryan.roberts@arm.com/ [3] https://lore.kernel.org/linux-mm/20230825135918.4164671-1-willy@infradead.org/ Thanks, Ryan Ryan Roberts (5): mm: Implement folio_remove_rmap_range() mm/mmu_gather: generalize mmu_gather rmap removal mechanism mm/mmu_gather: Remove encoded_page infrastructure mm: Refector release_pages() mm/mmu_gather: Store and process pages in contig ranges arch/s390/include/asm/tlb.h | 9 +- include/asm-generic/tlb.h | 49 ++++----- include/linux/mm.h | 11 +- include/linux/mm_types.h | 34 +----- include/linux/rmap.h | 2 + include/linux/swap.h | 6 +- mm/memory.c | 24 +++-- mm/mmu_gather.c | 114 ++++++++++++++------ mm/rmap.c | 125 ++++++++++++++++------ mm/swap.c | 201 ++++++++++++++++++++++-------------- mm/swap_state.c | 11 +- 11 files changed, 367 insertions(+), 219 deletions(-) --- 2.25.1