From patchwork Thu Feb 29 00:37:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13576145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15CACC5475B for ; Thu, 29 Feb 2024 00:38:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AE3F6B009E; Wed, 28 Feb 2024 19:38:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 95FD76B00A0; Wed, 28 Feb 2024 19:38:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FFAC6B00A1; Wed, 28 Feb 2024 19:38:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7085E6B009E for ; Wed, 28 Feb 2024 19:38:37 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EC1A01C1489 for ; Thu, 29 Feb 2024 00:38:36 +0000 (UTC) X-FDA: 81842980632.28.D6D3CB5 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf09.hostedemail.com (Postfix) with ESMTP id 1F082140008 for ; Thu, 29 Feb 2024 00:38:33 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mGqban28; spf=pass (imf09.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709167114; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aRxwZRNCH2mhvj5BH6CtV9uA6BSo86eped93k9fcnb8=; b=BGVyOyUIoO0oOtcjdaJO8u2uwB3JGHcXIQghSKSvfv7Dt2IYAYT3oLqksqYTfFWjA5pINC n2KxupLHLZzrOpOHzqw2IA7ognR/ocSOlk/QAwd5L1yDu2X2m7lkZFkHkLW6LTnPsTlzez dm1OfmeEoP7C1Xr6dOG+jwQtOzmwOcM= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=mGqban28; spf=pass (imf09.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709167114; a=rsa-sha256; cv=none; b=J6QYeJ5j43iWZqpOnvodlXTqlgEcMsgjenJwnAcghwT1fqNunwNqb0XkfiLcPtE0cknJ6J Zm0UQC6eDhdOpG/CsT9IO9QIEZnkbjKPo0cAJlr7ofTAbDq5uaQvinz7PM2FHfUI/747l8 Fota/XwaRecJfZ7Y1BfALbKFskF6tzw= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-6e48eef8be5so208276b3a.1 for ; Wed, 28 Feb 2024 16:38:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709167113; x=1709771913; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aRxwZRNCH2mhvj5BH6CtV9uA6BSo86eped93k9fcnb8=; b=mGqban28t1SKcyxcQAY/U7kcE27ryr9T4AVXjvpf9obNMf1xeBrq2vn84OudInHymQ vZ2A5exViuh6Q6OiaHfzu7pbK46gQz9WMKLftcGkhhkniMIpB2LP97umfPM1IknpEPca Zk8IaubUqrajXYn4RuEKHWLKLLrEK3TCdaX+Zwo3vP8hhDN98nr3Q4KTgxbX9R22leeE WPxADwRQJb6cGsukse3qQsRrgaVavp33X67TEthpNmcPHpelVmvdEapbB/S+HHSa3Z5n F1YmsmkeAfeHoHtLXIINjRVP/Rz7dbSmpdbzQrxYIXYO7qtwWcn0IP/0lyAVAkTm7Qog eugQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709167113; x=1709771913; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aRxwZRNCH2mhvj5BH6CtV9uA6BSo86eped93k9fcnb8=; b=qYvplgCayP6nPZHe7H5Fa138Qz/huUaJdWrZoPs0MKU7QeEM14iNjF025qU2vIP73N PGDBwcRsWns9WAIABWboDksF3SXngib62GdCNNg2qWbkRfh/cR+KZSP3b72DxLVARKZ6 3I7zXK45o6WWo16JWF3DTJDkxqwjp8mTpPXaD/exZFQOPto0zLiUy43pGN1LlFGc0kSO VaVUUjrYLByUFPeIE/G2V+rd65GLr/AlUQ5vEgW6Wz5r4/BjE185whMzWClLUv7nyp0P 7vEmKn2uNcXkX7Qqb9RmGvSogJpUyn7+oSP7M5Wr5v1S2BFkOxRxPuPGXhOrvYLe4fIc Y6Ew== X-Forwarded-Encrypted: i=1; AJvYcCXEza4UGwkuCmZ8OjHs4p0yUxH1umfJE4aLTqfSeh8UMWuppCOaop8miD0lYmEsuWIp+9GXMYgwJ885EFUXeBdvjQ8= X-Gm-Message-State: AOJu0Yzg0VG9fVjLWfXhv2Q7e7cdSyt1TXAd3eNU02y9raCg3isMirkH IcQgWa2HhNYShivWXTREuvrdVgM9hzOpG0/ADc7sVQh9TGIJANdS X-Google-Smtp-Source: AGHT+IFYesE9qHktmHaNdgA0gjlt042dxAyTHhmUPcBc/Iu8UblxtJZ6KbK4wn0XfTJXg38vPbuHew== X-Received: by 2002:a05:6a20:3252:b0:19f:f059:c190 with SMTP id hm18-20020a056a20325200b0019ff059c190mr905440pzc.24.1709167112788; Wed, 28 Feb 2024 16:38:32 -0800 (PST) Received: from localhost.localdomain ([2407:7000:8942:5500:5158:ed66:78b3:7fda]) by smtp.gmail.com with ESMTPSA id p3-20020a170902780300b001d9641003cfsm62647pll.142.2024.02.28.16.38.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Feb 2024 16:38:32 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org, ryan.roberts@arm.com, chrisl@kernel.org Cc: 21cnbao@gmail.com, linux-kernel@vger.kernel.org, mhocko@suse.com, shy828301@gmail.com, steven.price@arm.com, surenb@google.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yuzhao@google.com, kasong@tencent.com, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, hannes@cmpxchg.org, linux-arm-kernel@lists.infradead.org, Barry Song , Catalin Marinas , Will Deacon , Mark Rutland , Kemeng Shi , Anshuman Khandual , Peter Collingbourne , Peter Xu , Lorenzo Stoakes , "Mike Rapoport (IBM)" , Hugh Dickins , "Aneesh Kumar K.V" , Rick Edgecombe Subject: [PATCH RFC v2 1/5] arm64: mm: swap: support THP_SWAP on hardware with MTE Date: Thu, 29 Feb 2024 13:37:49 +1300 Message-Id: <20240229003753.134193-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240229003753.134193-1-21cnbao@gmail.com> References: <20240229003753.134193-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1F082140008 X-Rspam-User: X-Stat-Signature: 7pgi378r7e5t5paa9d7ery79k7amfz8g X-Rspamd-Server: rspam01 X-HE-Tag: 1709167113-117982 X-HE-Meta: U2FsdGVkX19+6uo2L78iZ+PQssg3OKIvLsFEJXjzwVpxTGIblX8dVfye5q8JdI7IBp9YPCERm4yPC2sEoLaZCigMc6D8itE9nxjF7RsM0xOBbSTLwXVsltNNisnpLEFay+/tAfWq4baTfTu5ox1i1Fb3S9AZEDtGFtQMBT6out//tNgV+un7FRwUvVPeZikH+vUZESU2y7QnHGgVrl/ShAUzsh5BAT0yfjGzT/y6TOoVRMxG3k7U0MT8q7wf8f00eSzMv137dp2d8UvNGNvmWvL7W3rThAIvVZY+MezzpY4McpOghNT/wxZQMLYhN+LG67WEpNlHKpFe6yczAqJ9t+x4XBjjGuO16eOcQ4VofQ+AHKAEIPpvK/oEDORBTIueY+ac6BRfdLwjuW7j1gwTdxCOxi6bUQD7kJsItdMA0+jblEP52Yxeby/+3qvTRALO0b5+SUr3XDmbFS2DqBZYVwbji7azwJ0su4KOOmA6gFeOMZKoOXeO0wa/d2i9F176uE5kXP8xaFxjvAzZhzIK/nfm4t4nXZE72lKULUaWQf/CH8ifnQunpSkS12WebLblCzAZySywGKH6MoSlpl0ITCozJX2O6aEndltqkxyDhLiZgRZNgRX3/KucbejQlIu6B/CtXIUj+wv7wDdEFbJ/N00ZNhxpiwI8K0A5as6K6GUHJ1nfCpBJD47DoVtLDjy6uDtY0YDGNkt4q6HGfJmgwB83vNP4u/8NaHBtu/h0sNDTMQaouxvC1ehmxJWuuKziCLtUL4SOd+DUkPF4wiWSSVMqXWrxqPMzFk/WvIFgNfXRNsFa95koOtuxRWV9rE+FdRqC1NtVGjKV/kYBbAyEmEvV6uOxpUL/LddSHfEUpmV34S7oO+XUB62FzBAWn34RtX6hHypG8RwpjAuk3K4qEAYm+18XwYB493aerVOqwMvK/gdeNreWvG2sLnOSGhe2ZocHeodYtCkd/gui1Id rgdNUQip WhlabyFo+iQRecc2UdAaQNOwxrCQjY0flpdeWWjTqeyb21gFGzOM/xsaI1usLffjYOX/zujGI0jsgh9On5ulHGf/3imx/ttEMM5VxNVe1lPhEGHiti09dusaH94lKZEPJZf89k/pD2RBZbnY0nlgpHnC5W1yGEKqAfP3L X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Commit d0637c505f8a1 ("arm64: enable THP_SWAP for arm64") brings up THP_SWAP on ARM64, but it doesn't enable THP_SWP on hardware with MTE as the MTE code works with the assumption tags save/restore is always handling a folio with only one page. The limitation should be removed as more and more ARM64 SoCs have this feature. Co-existence of MTE and THP_SWAP becomes more and more important. This patch makes MTE tags saving support large folios, then we don't need to split large folios into base pages for swapping out on ARM64 SoCs with MTE any more. arch_prepare_to_swap() should take folio rather than page as parameter because we support THP swap-out as a whole. It saves tags for all pages in a large folio. As now we are restoring tags based-on folio, in arch_swap_restore(), we may increase some extra loops and early-exitings while refaulting a large folio which is still in swapcache in do_swap_page(). In case a large folio has nr pages, do_swap_page() will only set the PTE of the particular page which is causing the page fault. Thus do_swap_page() runs nr times, and each time, arch_swap_restore() will loop nr times for those subpages in the folio. So right now the algorithmic complexity becomes O(nr^2). Once we support mapping large folios in do_swap_page(), extra loops and early-exitings will decrease while not being completely removed as a large folio might get partially tagged in corner cases such as, 1. a large folio in swapcache can be partially unmapped, thus, MTE tags for the unmapped pages will be invalidated; 2. users might use mprotect() to set MTEs on a part of a large folio. arch_thp_swp_supported() is dropped since ARM64 MTE was the only one who needed it. Cc: Catalin Marinas Cc: Will Deacon Cc: Ryan Roberts Cc: Mark Rutland Cc: David Hildenbrand Cc: Kemeng Shi Cc: "Matthew Wilcox (Oracle)" Cc: Anshuman Khandual Cc: Peter Collingbourne Cc: Steven Price Cc: Yosry Ahmed Cc: Peter Xu Cc: Lorenzo Stoakes Cc: "Mike Rapoport (IBM)" Cc: Hugh Dickins CC: "Aneesh Kumar K.V" Cc: Rick Edgecombe Signed-off-by: Barry Song Reviewed-by: Steven Price Acked-by: Chris Li --- arch/arm64/include/asm/pgtable.h | 19 ++------------ arch/arm64/mm/mteswap.c | 43 ++++++++++++++++++++++++++++++++ include/linux/huge_mm.h | 12 --------- include/linux/pgtable.h | 2 +- mm/page_io.c | 2 +- mm/swap_slots.c | 2 +- 6 files changed, 48 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 401087e8a43d..7a54750770b8 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -45,12 +45,6 @@ __flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1) #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -static inline bool arch_thp_swp_supported(void) -{ - return !system_supports_mte(); -} -#define arch_thp_swp_supported arch_thp_swp_supported - /* * Outside of a few very special situations (e.g. hibernation), we always * use broadcast TLB invalidation instructions, therefore a spurious page @@ -1095,12 +1089,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, #ifdef CONFIG_ARM64_MTE #define __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap(struct page *page) -{ - if (system_supports_mte()) - return mte_save_tags(page); - return 0; -} +extern int arch_prepare_to_swap(struct folio *folio); #define __HAVE_ARCH_SWAP_INVALIDATE static inline void arch_swap_invalidate_page(int type, pgoff_t offset) @@ -1116,11 +1105,7 @@ static inline void arch_swap_invalidate_area(int type) } #define __HAVE_ARCH_SWAP_RESTORE -static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) -{ - if (system_supports_mte()) - mte_restore_tags(entry, &folio->page); -} +extern void arch_swap_restore(swp_entry_t entry, struct folio *folio); #endif /* CONFIG_ARM64_MTE */ diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index a31833e3ddc5..295836fef620 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -68,6 +68,13 @@ void mte_invalidate_tags(int type, pgoff_t offset) mte_free_tag_storage(tags); } +static inline void __mte_invalidate_tags(struct page *page) +{ + swp_entry_t entry = page_swap_entry(page); + + mte_invalidate_tags(swp_type(entry), swp_offset(entry)); +} + void mte_invalidate_tags_area(int type) { swp_entry_t entry = swp_entry(type, 0); @@ -83,3 +90,39 @@ void mte_invalidate_tags_area(int type) } xa_unlock(&mte_pages); } + +int arch_prepare_to_swap(struct folio *folio) +{ + long i, nr; + int err; + + if (!system_supports_mte()) + return 0; + + nr = folio_nr_pages(folio); + + for (i = 0; i < nr; i++) { + err = mte_save_tags(folio_page(folio, i)); + if (err) + goto out; + } + return 0; + +out: + while (i--) + __mte_invalidate_tags(folio_page(folio, i)); + return err; +} + +void arch_swap_restore(swp_entry_t entry, struct folio *folio) +{ + if (system_supports_mte()) { + long i, nr = folio_nr_pages(folio); + + entry.val -= swp_offset(entry) & (nr - 1); + for (i = 0; i < nr; i++) { + mte_restore_tags(entry, folio_page(folio, i)); + entry.val++; + } + } +} diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index de0c89105076..e04b93c43965 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -535,16 +535,4 @@ static inline int split_folio_to_order(struct folio *folio, int new_order) #define split_folio_to_list(f, l) split_folio_to_list_to_order(f, l, 0) #define split_folio(f) split_folio_to_order(f, 0) -/* - * archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to - * limitations in the implementation like arm64 MTE can override this to - * false - */ -#ifndef arch_thp_swp_supported -static inline bool arch_thp_swp_supported(void) -{ - return true; -} -#endif - #endif /* _LINUX_HUGE_MM_H */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a36cf4e124b0..ec7efce0f3f0 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1052,7 +1052,7 @@ static inline int arch_unmap_one(struct mm_struct *mm, * prototypes must be defined in the arch-specific asm/pgtable.h file. */ #ifndef __HAVE_ARCH_PREPARE_TO_SWAP -static inline int arch_prepare_to_swap(struct page *page) +static inline int arch_prepare_to_swap(struct folio *folio) { return 0; } diff --git a/mm/page_io.c b/mm/page_io.c index ae2b49055e43..a9a7c236aecc 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -189,7 +189,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc) * Arch code may have to preserve more data than just the page * contents, e.g. memory tags. */ - ret = arch_prepare_to_swap(&folio->page); + ret = arch_prepare_to_swap(folio); if (ret) { folio_mark_dirty(folio); folio_unlock(folio); diff --git a/mm/swap_slots.c b/mm/swap_slots.c index 90973ce7881d..53abeaf1371d 100644 --- a/mm/swap_slots.c +++ b/mm/swap_slots.c @@ -310,7 +310,7 @@ swp_entry_t folio_alloc_swap(struct folio *folio) entry.val = 0; if (folio_test_large(folio)) { - if (IS_ENABLED(CONFIG_THP_SWAP) && arch_thp_swp_supported()) + if (IS_ENABLED(CONFIG_THP_SWAP)) get_swap_pages(1, &entry, folio_nr_pages(folio)); goto out; }