From patchwork Sun Nov 19 16:56:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 12525C54FB9 for ; Sun, 19 Nov 2023 16:58:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kH+gQbyIJPtG+HyaXIXe2e3wfeTxYbLpIQoXjW0TXPU=; b=48s6Y1X0V2tyTA o0m/zPgxXG1voWd4+dTQZLqIE1Kw2dyHMysrcultBMocVy3RAloHiz4aNZ+SIVYyx43JbnlAHZClM bOmSEODbBVvld+6IeRlem7w0Uss7Q9g3yEzyUpXo/aNNLn5ueGuFbLGKEXC8WAVV11UfxBCC8uTH6 85ekyvmYk3rInmnatgu44zjCTmndW4YSKjQEtlXBn+GU+je+MRxR8WBgY18e9wvGcxsW0MmFqd80D fE98JWbUz15ysLPTiluiLvQVd+lzttnmklDt2Lr90OIiGEDDP+eMRA8gqyxDcnfT1tS81q/nECsId reyUI2LJXK/3fMOdOXaw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7J-00AQFZ-2o; Sun, 19 Nov 2023 16:57:54 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7E-00AQC1-16 for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:57:50 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2411FFEC; Sun, 19 Nov 2023 08:58:31 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CC6C63F6C4; Sun, 19 Nov 2023 08:57:39 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 01/27] arm64: mte: Rework naming for tag manipulation functions Date: Sun, 19 Nov 2023 16:56:55 +0000 Message-Id: <20231119165721.9849-2-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085748_470316_0613C4B4 X-CRM114-Status: GOOD ( 26.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The tag save/restore/copy functions could be more explicit about from where the tags are coming from and where they are being copied to. Renaming the functions to make it easier to understand what they are doing: - Rename the mte_clear_page_tags() 'addr' parameter to 'page_addr', to match the other functions that take a page address as parameter. - Rename mte_save/restore_tags() to mte_save/restore_page_tags_by_swp_entry() to make it clear that they are saved in a collection indexed by swp_entry (this will become important when they will be also saved in a collection indexed by page pfn). Same applies to mte_invalidate_tags{,_area}_by_swp_entry(). - Rename mte_save/restore_page_tags() to make it clear where the tags are going to be saved, respectively from where they are restored - in a previously allocated memory buffer, not in an xarray, like when the tags are saved when swapping. Rename the action to 'copy' instead of 'save'/'restore' to match the copy from user functions, which also copy tags to memory. - Rename mte_allocate/free_tag_storage() to mte_allocate/free_tag_buf() to make it clear the functions have nothing to do with the memory where the corresponding tags for a page live. Change the parameter type for mte_free_tag_buf()) to be void *, to match the return value of mte_allocate_tag_buf(). Also do that because that memory is opaque and it is not meant to be directly deferenced. In the name of consistency rename local variables from tag_storage to tags. Give a similar treatment to the hibernation code that saves and restores the tags for all tagged pages. In the same spirit, rename MTE_PAGE_TAG_STORAGE to MTE_PAGE_TAG_STORAGE_SIZE to make it clear that it relates to the size of the memory needed to save the tags for a page. Oportunistically rename MTE_TAG_SIZE to MTE_TAG_SIZE_BITS to make it clear it is measured in bits, not bytes, like the rest of the size variable from the same header file. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte-def.h | 16 +++++----- arch/arm64/include/asm/mte.h | 23 +++++++++------ arch/arm64/include/asm/pgtable.h | 8 ++--- arch/arm64/kernel/elfcore.c | 14 ++++----- arch/arm64/kernel/hibernate.c | 46 ++++++++++++++--------------- arch/arm64/lib/mte.S | 18 ++++++------ arch/arm64/mm/mteswap.c | 50 ++++++++++++++++---------------- 7 files changed, 90 insertions(+), 85 deletions(-) diff --git a/arch/arm64/include/asm/mte-def.h b/arch/arm64/include/asm/mte-def.h index 14ee86b019c2..eb0d76a6bdcf 100644 --- a/arch/arm64/include/asm/mte-def.h +++ b/arch/arm64/include/asm/mte-def.h @@ -5,14 +5,14 @@ #ifndef __ASM_MTE_DEF_H #define __ASM_MTE_DEF_H -#define MTE_GRANULE_SIZE UL(16) -#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1)) -#define MTE_GRANULES_PER_PAGE (PAGE_SIZE / MTE_GRANULE_SIZE) -#define MTE_TAG_SHIFT 56 -#define MTE_TAG_SIZE 4 -#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE - 1)), MTE_TAG_SHIFT) -#define MTE_PAGE_TAG_STORAGE (MTE_GRANULES_PER_PAGE * MTE_TAG_SIZE / 8) +#define MTE_GRANULE_SIZE UL(16) +#define MTE_GRANULE_MASK (~(MTE_GRANULE_SIZE - 1)) +#define MTE_GRANULES_PER_PAGE (PAGE_SIZE / MTE_GRANULE_SIZE) +#define MTE_TAG_SHIFT 56 +#define MTE_TAG_SIZE_BITS 4 +#define MTE_TAG_MASK GENMASK((MTE_TAG_SHIFT + (MTE_TAG_SIZE_BITS - 1)), MTE_TAG_SHIFT) +#define MTE_PAGE_TAG_STORAGE_SIZE (MTE_GRANULES_PER_PAGE * MTE_TAG_SIZE_BITS / 8) -#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" +#define __MTE_PREAMBLE ARM64_ASM_PREAMBLE ".arch_extension memtag\n" #endif /* __ASM_MTE_DEF_H */ diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 91fbd5c8a391..8034695b3dd7 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -18,19 +18,24 @@ #include -void mte_clear_page_tags(void *addr); +void mte_clear_page_tags(void *page_addr); + unsigned long mte_copy_tags_from_user(void *to, const void __user *from, unsigned long n); unsigned long mte_copy_tags_to_user(void __user *to, void *from, unsigned long n); -int mte_save_tags(struct page *page); -void mte_save_page_tags(const void *page_addr, void *tag_storage); -void mte_restore_tags(swp_entry_t entry, struct page *page); -void mte_restore_page_tags(void *page_addr, const void *tag_storage); -void mte_invalidate_tags(int type, pgoff_t offset); -void mte_invalidate_tags_area(int type); -void *mte_allocate_tag_storage(void); -void mte_free_tag_storage(char *storage); + +int mte_save_page_tags_by_swp_entry(struct page *page); +void mte_restore_page_tags_by_swp_entry(swp_entry_t entry, struct page *page); + +void mte_copy_page_tags_to_buf(const void *page_addr, void *to); +void mte_copy_page_tags_from_buf(void *page_addr, const void *from); + +void mte_invalidate_tags_by_swp_entry(int type, pgoff_t offset); +void mte_invalidate_tags_area_by_swp_entry(int type); + +void *mte_allocate_tag_buf(void); +void mte_free_tag_buf(void *buf); #ifdef CONFIG_ARM64_MTE diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index b19a8aee684c..9b32c74b4a1b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1039,7 +1039,7 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, static inline int arch_prepare_to_swap(struct page *page) { if (system_supports_mte()) - return mte_save_tags(page); + return mte_save_page_tags_by_swp_entry(page); return 0; } @@ -1047,20 +1047,20 @@ static inline int arch_prepare_to_swap(struct page *page) static inline void arch_swap_invalidate_page(int type, pgoff_t offset) { if (system_supports_mte()) - mte_invalidate_tags(type, offset); + mte_invalidate_tags_by_swp_entry(type, offset); } static inline void arch_swap_invalidate_area(int type) { if (system_supports_mte()) - mte_invalidate_tags_area(type); + mte_invalidate_tags_area_by_swp_entry(type); } #define __HAVE_ARCH_SWAP_RESTORE static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) { if (system_supports_mte()) - mte_restore_tags(entry, &folio->page); + mte_restore_page_tags_by_swp_entry(entry, &folio->page); } #endif /* CONFIG_ARM64_MTE */ diff --git a/arch/arm64/kernel/elfcore.c b/arch/arm64/kernel/elfcore.c index 2e94d20c4ac7..e9ae00dacad8 100644 --- a/arch/arm64/kernel/elfcore.c +++ b/arch/arm64/kernel/elfcore.c @@ -17,7 +17,7 @@ static unsigned long mte_vma_tag_dump_size(struct core_vma_metadata *m) { - return (m->dump_size >> PAGE_SHIFT) * MTE_PAGE_TAG_STORAGE; + return (m->dump_size >> PAGE_SHIFT) * MTE_PAGE_TAG_STORAGE_SIZE; } /* Derived from dump_user_range(); start/end must be page-aligned */ @@ -38,7 +38,7 @@ static int mte_dump_tag_range(struct coredump_params *cprm, * have been all zeros. */ if (!page) { - dump_skip(cprm, MTE_PAGE_TAG_STORAGE); + dump_skip(cprm, MTE_PAGE_TAG_STORAGE_SIZE); continue; } @@ -48,12 +48,12 @@ static int mte_dump_tag_range(struct coredump_params *cprm, */ if (!page_mte_tagged(page)) { put_page(page); - dump_skip(cprm, MTE_PAGE_TAG_STORAGE); + dump_skip(cprm, MTE_PAGE_TAG_STORAGE_SIZE); continue; } if (!tags) { - tags = mte_allocate_tag_storage(); + tags = mte_allocate_tag_buf(); if (!tags) { put_page(page); ret = 0; @@ -61,16 +61,16 @@ static int mte_dump_tag_range(struct coredump_params *cprm, } } - mte_save_page_tags(page_address(page), tags); + mte_copy_page_tags_to_buf(page_address(page), tags); put_page(page); - if (!dump_emit(cprm, tags, MTE_PAGE_TAG_STORAGE)) { + if (!dump_emit(cprm, tags, MTE_PAGE_TAG_STORAGE_SIZE)) { ret = 0; break; } } if (tags) - mte_free_tag_storage(tags); + mte_free_tag_buf(tags); return ret; } diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 02870beb271e..a3b0e7b32457 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -215,41 +215,41 @@ static int create_safe_exec_page(void *src_start, size_t length, #ifdef CONFIG_ARM64_MTE -static DEFINE_XARRAY(mte_pages); +static DEFINE_XARRAY(tags_by_pfn); -static int save_tags(struct page *page, unsigned long pfn) +static int save_page_tags_by_pfn(struct page *page, unsigned long pfn) { - void *tag_storage, *ret; + void *tags, *ret; - tag_storage = mte_allocate_tag_storage(); - if (!tag_storage) + tags = mte_allocate_tag_buf(); + if (!tags) return -ENOMEM; - mte_save_page_tags(page_address(page), tag_storage); + mte_copy_page_tags_to_buf(page_address(page), tags); - ret = xa_store(&mte_pages, pfn, tag_storage, GFP_KERNEL); + ret = xa_store(&tags_by_pfn, pfn, tags, GFP_KERNEL); if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { - mte_free_tag_storage(tag_storage); + mte_free_tag_buf(tags); return xa_err(ret); } else if (WARN(ret, "swsusp: %s: Duplicate entry", __func__)) { - mte_free_tag_storage(ret); + mte_free_tag_buf(ret); } return 0; } -static void swsusp_mte_free_storage(void) +static void swsusp_mte_free_tags(void) { - XA_STATE(xa_state, &mte_pages, 0); + XA_STATE(xa_state, &tags_by_pfn, 0); void *tags; - xa_lock(&mte_pages); + xa_lock(&tags_by_pfn); xas_for_each(&xa_state, tags, ULONG_MAX) { - mte_free_tag_storage(tags); + mte_free_tag_buf(tags); } - xa_unlock(&mte_pages); + xa_unlock(&tags_by_pfn); - xa_destroy(&mte_pages); + xa_destroy(&tags_by_pfn); } static int swsusp_mte_save_tags(void) @@ -273,9 +273,9 @@ static int swsusp_mte_save_tags(void) if (!page_mte_tagged(page)) continue; - ret = save_tags(page, pfn); + ret = save_page_tags_by_pfn(page, pfn); if (ret) { - swsusp_mte_free_storage(); + swsusp_mte_free_tags(); goto out; } @@ -290,25 +290,25 @@ static int swsusp_mte_save_tags(void) static void swsusp_mte_restore_tags(void) { - XA_STATE(xa_state, &mte_pages, 0); + XA_STATE(xa_state, &tags_by_pfn, 0); int n = 0; void *tags; - xa_lock(&mte_pages); + xa_lock(&tags_by_pfn); xas_for_each(&xa_state, tags, ULONG_MAX) { unsigned long pfn = xa_state.xa_index; struct page *page = pfn_to_online_page(pfn); - mte_restore_page_tags(page_address(page), tags); + mte_copy_page_tags_from_buf(page_address(page), tags); - mte_free_tag_storage(tags); + mte_free_tag_buf(tags); n++; } - xa_unlock(&mte_pages); + xa_unlock(&tags_by_pfn); pr_info("Restored %d MTE pages\n", n); - xa_destroy(&mte_pages); + xa_destroy(&tags_by_pfn); } #else /* CONFIG_ARM64_MTE */ diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 5018ac03b6bf..9f623e9da09f 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -119,7 +119,7 @@ SYM_FUNC_START(mte_copy_tags_to_user) cbz x2, 2f 1: ldg x4, [x1] - ubfx x4, x4, #MTE_TAG_SHIFT, #MTE_TAG_SIZE + ubfx x4, x4, #MTE_TAG_SHIFT, #MTE_TAG_SIZE_BITS USER(2f, sttrb w4, [x0]) add x0, x0, #1 add x1, x1, #MTE_GRANULE_SIZE @@ -132,11 +132,11 @@ USER(2f, sttrb w4, [x0]) SYM_FUNC_END(mte_copy_tags_to_user) /* - * Save the tags in a page + * Copy the tags in a page to a buffer * x0 - page address - * x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes + * x1 - memory buffer, MTE_PAGE_TAG_STORAGE_SIZE bytes */ -SYM_FUNC_START(mte_save_page_tags) +SYM_FUNC_START(mte_copy_page_tags_to_buf) multitag_transfer_size x7, x5 1: mov x2, #0 @@ -153,14 +153,14 @@ SYM_FUNC_START(mte_save_page_tags) b.ne 1b ret -SYM_FUNC_END(mte_save_page_tags) +SYM_FUNC_END(mte_copy_page_tags_to_buf) /* - * Restore the tags in a page + * Restore the tags in a page from a buffer * x0 - page address - * x1 - tag storage, MTE_PAGE_TAG_STORAGE bytes + * x1 - memory buffer, MTE_PAGE_TAG_STORAGE_SIZE bytes */ -SYM_FUNC_START(mte_restore_page_tags) +SYM_FUNC_START(mte_copy_page_tags_from_buf) multitag_transfer_size x7, x5 1: ldr x2, [x1], #8 @@ -174,4 +174,4 @@ SYM_FUNC_START(mte_restore_page_tags) b.ne 1b ret -SYM_FUNC_END(mte_restore_page_tags) +SYM_FUNC_END(mte_copy_page_tags_from_buf) diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index a31833e3ddc5..2a43746b803f 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -7,79 +7,79 @@ #include #include -static DEFINE_XARRAY(mte_pages); +static DEFINE_XARRAY(tags_by_swp_entry); -void *mte_allocate_tag_storage(void) +void *mte_allocate_tag_buf(void) { /* tags granule is 16 bytes, 2 tags stored per byte */ - return kmalloc(MTE_PAGE_TAG_STORAGE, GFP_KERNEL); + return kmalloc(MTE_PAGE_TAG_STORAGE_SIZE, GFP_KERNEL); } -void mte_free_tag_storage(char *storage) +void mte_free_tag_buf(void *buf) { - kfree(storage); + kfree(buf); } -int mte_save_tags(struct page *page) +int mte_save_page_tags_by_swp_entry(struct page *page) { - void *tag_storage, *ret; + void *tags, *ret; if (!page_mte_tagged(page)) return 0; - tag_storage = mte_allocate_tag_storage(); - if (!tag_storage) + tags = mte_allocate_tag_buf(); + if (!tags) return -ENOMEM; - mte_save_page_tags(page_address(page), tag_storage); + mte_copy_page_tags_to_buf(page_address(page), tags); /* lookup the swap entry.val from the page */ - ret = xa_store(&mte_pages, page_swap_entry(page).val, tag_storage, + ret = xa_store(&tags_by_swp_entry, page_swap_entry(page).val, tags, GFP_KERNEL); if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { - mte_free_tag_storage(tag_storage); + mte_free_tag_buf(tags); return xa_err(ret); } else if (ret) { /* Entry is being replaced, free the old entry */ - mte_free_tag_storage(ret); + mte_free_tag_buf(ret); } return 0; } -void mte_restore_tags(swp_entry_t entry, struct page *page) +void mte_restore_page_tags_by_swp_entry(swp_entry_t entry, struct page *page) { - void *tags = xa_load(&mte_pages, entry.val); + void *tags = xa_load(&tags_by_swp_entry, entry.val); if (!tags) return; if (try_page_mte_tagging(page)) { - mte_restore_page_tags(page_address(page), tags); + mte_copy_page_tags_from_buf(page_address(page), tags); set_page_mte_tagged(page); } } -void mte_invalidate_tags(int type, pgoff_t offset) +void mte_invalidate_tags_by_swp_entry(int type, pgoff_t offset) { swp_entry_t entry = swp_entry(type, offset); - void *tags = xa_erase(&mte_pages, entry.val); + void *tags = xa_erase(&tags_by_swp_entry, entry.val); - mte_free_tag_storage(tags); + mte_free_tag_buf(tags); } -void mte_invalidate_tags_area(int type) +void mte_invalidate_tags_area_by_swp_entry(int type) { swp_entry_t entry = swp_entry(type, 0); swp_entry_t last_entry = swp_entry(type + 1, 0); void *tags; - XA_STATE(xa_state, &mte_pages, entry.val); + XA_STATE(xa_state, &tags_by_swp_entry, entry.val); - xa_lock(&mte_pages); + xa_lock(&tags_by_swp_entry); xas_for_each(&xa_state, tags, last_entry.val - 1) { - __xa_erase(&mte_pages, xa_state.xa_index); - mte_free_tag_storage(tags); + __xa_erase(&tags_by_swp_entry, xa_state.xa_index); + mte_free_tag_buf(tags); } - xa_unlock(&mte_pages); + xa_unlock(&tags_by_swp_entry); } From patchwork Sun Nov 19 16:56:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CDD67C54E76 for ; Sun, 19 Nov 2023 16:58:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=M7JRDyquJEJxojuI8UAThkMApZ9BMI0dodcK8cEJhKc=; b=A8Ka3bAQyHHbWS G4hg6jVrTgu4t6VXQx47sPFY83QDCiODWHXatW+fIO4dXyFjOps9HdZvJKzuYaVvXGkjmBr26c2Wo LCdDMBdLaKqusomBrWjlZASfYccLOx078Ez8DY/Zfs7EeauFFPybQRlhh+rUMKWFLzMZnfQvR8XGV Z0Or2oEKeLn06BpwbXuHBzqPvMCCiLJhdtPof+GhKqZrDpOKvCKVPCzRNEMAuDaK4EDbHLpHBSLiU OYPW+MibXaV7AE1NuyUiD5rMUxUwHOkFvnCaoTVGaN8Kp+eIYJwTCP7ZoEtZ/ozRGM1F2yPMmYzAK lGwTthaUSqZQ8Q8OdNqg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7R-00AQLt-2D; Sun, 19 Nov 2023 16:58:01 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7H-00AQD2-2D for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:57:53 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5DFD71042; Sun, 19 Nov 2023 08:58:36 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2F2F53F6C4; Sun, 19 Nov 2023 08:57:45 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 02/27] arm64: mte: Rename __GFP_ZEROTAGS to __GFP_TAGGED Date: Sun, 19 Nov 2023 16:56:56 +0000 Message-Id: <20231119165721.9849-3-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085752_068697_AD6D69E6 X-CRM114-Status: GOOD ( 17.80 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org __GFP_ZEROTAGS is used to instruct the page allocator to zero the tags at the same time as the physical frame is zeroed. The name can be slightly misleading, because it doesn't mean that the code will zero the tags unconditionally, but that the tags will be zeroed if and only if the physical frame is also zeroed (either __GFP_ZERO is set or init_on_alloc is 1). Rename it to __GFP_TAGGED, in preparation for it to be used by the page allocator to recognize when an allocation is tagged (has metadata). Signed-off-by: Alexandru Elisei --- arch/arm64/mm/fault.c | 2 +- include/linux/gfp_types.h | 14 +++++++------- include/trace/events/mmflags.h | 2 +- mm/page_alloc.c | 2 +- 4 files changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 460d799e1296..daa91608d917 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -948,7 +948,7 @@ struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, * separate DC ZVA and STGM. */ if (vma->vm_flags & VM_MTE) - flags |= __GFP_ZEROTAGS; + flags |= __GFP_TAGGED; return vma_alloc_folio(flags, 0, vma, vaddr, false); } diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index 6583a58670c5..37b9e265d77e 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -45,7 +45,7 @@ typedef unsigned int __bitwise gfp_t; #define ___GFP_HARDWALL 0x100000u #define ___GFP_THISNODE 0x200000u #define ___GFP_ACCOUNT 0x400000u -#define ___GFP_ZEROTAGS 0x800000u +#define ___GFP_TAGGED 0x800000u #ifdef CONFIG_KASAN_HW_TAGS #define ___GFP_SKIP_ZERO 0x1000000u #define ___GFP_SKIP_KASAN 0x2000000u @@ -226,11 +226,11 @@ typedef unsigned int __bitwise gfp_t; * * %__GFP_ZERO returns a zeroed page on success. * - * %__GFP_ZEROTAGS zeroes memory tags at allocation time if the memory itself - * is being zeroed (either via __GFP_ZERO or via init_on_alloc, provided that - * __GFP_SKIP_ZERO is not set). This flag is intended for optimization: setting - * memory tags at the same time as zeroing memory has minimal additional - * performace impact. + * %__GFP_TAGGED marks the allocation as having tags, which will be zeroed it + * allocation time if the memory itself is being zeroed (either via __GFP_ZERO + * or via init_on_alloc, provided that __GFP_SKIP_ZERO is not set). This flag is + * intended for optimization: setting memory tags at the same time as zeroing + * memory has minimal additional performace impact. * * %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation. * Used for userspace and vmalloc pages; the latter are unpoisoned by @@ -241,7 +241,7 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) #define __GFP_COMP ((__force gfp_t)___GFP_COMP) #define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) -#define __GFP_ZEROTAGS ((__force gfp_t)___GFP_ZEROTAGS) +#define __GFP_TAGGED ((__force gfp_t)___GFP_TAGGED) #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO) #define __GFP_SKIP_KASAN ((__force gfp_t)___GFP_SKIP_KASAN) diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index d801409b33cf..6ca0d5ed46c0 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -50,7 +50,7 @@ gfpflag_string(__GFP_RECLAIM), \ gfpflag_string(__GFP_DIRECT_RECLAIM), \ gfpflag_string(__GFP_KSWAPD_RECLAIM), \ - gfpflag_string(__GFP_ZEROTAGS) + gfpflag_string(__GFP_TAGGED) #ifdef CONFIG_KASAN_HW_TAGS #define __def_gfpflag_names_kasan , \ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 733732e7e0ba..770e585b77c8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1483,7 +1483,7 @@ inline void post_alloc_hook(struct page *page, unsigned int order, { bool init = !want_init_on_free() && want_init_on_alloc(gfp_flags) && !should_skip_init(gfp_flags); - bool zero_tags = init && (gfp_flags & __GFP_ZEROTAGS); + bool zero_tags = init && (gfp_flags & __GFP_TAGGED); int i; set_page_private(page, 0); From patchwork Sun Nov 19 16:56:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460590 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1323BC072A2 for ; Sun, 19 Nov 2023 16:58:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=uPLtYB7a51CeDFQgowfoNe/bKrZ2dg7ClhuxBU5+9V4=; b=PHPof++ANtuK0U F8exjPVYi6JOb5P+HJlHw3bBQZ23JbjpEAZ9RZ1pqS6rO8I5BtFOvi8maKY2LjASdjF8wxzPB9WVz LA+OtxgGxzSyni3XbhZusDVhppXmmwAKaWLEAutSyC74VzqTkn5PM2esZrV235jxSgyaCS16Ukpg0 BEr7MkRvi4cW6vAsQS2MCBV24ORYsq9974GVO2INEj+HC/XoG88xe3GCl2+SYSTjmbSaCWI10oMBp 5tca/F30HKgpNxVivMp8i4nukcaoCzIgfbbuip/I+/PwB/jxe2LLXQn3XdpjuIqIn/k4kEiwVtVBn 2AQhQIJLandyTYbaxMVg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7T-00AQO3-1i; Sun, 19 Nov 2023 16:58:03 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7M-00AQGv-0A for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:57:59 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 936051476; Sun, 19 Nov 2023 08:58:41 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 64E5A3F6C4; Sun, 19 Nov 2023 08:57:50 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 03/27] mm: cma: Make CMA_ALLOC_SUCCESS/FAIL count the number of pages Date: Sun, 19 Nov 2023 16:56:57 +0000 Message-Id: <20231119165721.9849-4-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085756_258718_F70E7398 X-CRM114-Status: GOOD ( 12.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The CMA_ALLOC_SUCCESS, respectively CMA_ALLOC_FAIL, are increased by one after each cma_alloc() function call. This is done even though cma_alloc() can allocate an arbitrary number of CMA pages. When looking at /proc/vmstat, the number of successful (or failed) cma_alloc() calls doesn't tell much with regards to how many CMA pages were allocated via cma_alloc() versus via the page allocator (regular allocation request or PCP lists refill). This can also be rather confusing to a user who isn't familiar with the code, since the unit of measurement for nr_free_cma is the number of pages, but cma_alloc_success and cma_alloc_fail count the number of cma_alloc() function calls. Let's make this consistent, and arguably more useful, by having CMA_ALLOC_SUCCESS count the number of successfully allocated CMA pages, and CMA_ALLOC_FAIL count the number of pages cma_alloc() failed to allocate. For users that wish to track the number of cma_alloc() calls, there are tracepoints for that already implemented. Signed-off-by: Alexandru Elisei --- mm/cma.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/cma.c b/mm/cma.c index 2b2494fd6b59..2b74db5116d5 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -517,10 +517,10 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, pr_debug("%s(): returned %p\n", __func__, page); out: if (page) { - count_vm_event(CMA_ALLOC_SUCCESS); + count_vm_events(CMA_ALLOC_SUCCESS, count); cma_sysfs_account_success_pages(cma, count); } else { - count_vm_event(CMA_ALLOC_FAIL); + count_vm_events(CMA_ALLOC_FAIL, count); if (cma) cma_sysfs_account_fail_pages(cma, count); } From patchwork Sun Nov 19 16:56:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94A06C5AE5B for ; Sun, 19 Nov 2023 16:58:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hp2HJx7FD19aLEyZJZ6wByo+HXQimWXr2aRvvjOV+bk=; b=04TvAm7PMpe25i YICtNkXmc3ZVDpr2eppOJIRfPCz8yGGDmy+hZY2/5PNDmA1RNS5z8dlGD/eYgJZW2ovGhqdVzZKRp mgN+PQsg9G8KPfHXWY5H5PjOxdaFx+DQ4BKTQywUnrH8qsV37mN86aNXsc/vhCqK9UPF1gxX+gWMG rSifaHPtKMf72icItarBn3uQLuyiFVRLbiV4s0VoeOLtvfxVBNkzCeyqsR2xs0V0XSKQl/rjaZ88B kpfvJR8HsuxKGHtUTEYeyg1mofMcBpyQQrA9F9Sd+/MgHvT6m+JPb9CiwVwlM9IKFfCNzIqzI3L2+ MDW5jnbMbOw2JFBTWh1w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7a-00AQU3-1f; Sun, 19 Nov 2023 16:58:10 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7R-00AQLJ-0x for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:02 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CD2BD1007; Sun, 19 Nov 2023 08:58:46 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9ACFC3F6C4; Sun, 19 Nov 2023 08:57:55 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 04/27] mm: migrate/mempolicy: Add hook to modify migration target gfp Date: Sun, 19 Nov 2023 16:56:58 +0000 Message-Id: <20231119165721.9849-5-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085801_435010_DA321D09 X-CRM114-Status: GOOD ( 12.12 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org It might be desirable for an architecture to modify the gfp flags used to allocate the destination page for migration based on the page that it is being replaced. For example, if an architectures has metadata associated with a page (like arm64, when the memory tagging extension is implemented), it can request that the destination page similarly has storage for tags already allocated. No functional change. Signed-off-by: Alexandru Elisei --- include/linux/migrate.h | 4 ++++ mm/mempolicy.c | 2 ++ mm/migrate.c | 3 +++ 3 files changed, 9 insertions(+) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 2ce13e8a309b..0acef592043c 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -60,6 +60,10 @@ struct movable_operations { /* Defined in mm/debug.c: */ extern const char *migrate_reason_names[MR_TYPES]; +#ifndef arch_migration_target_gfp +#define arch_migration_target_gfp(src, gfp) 0 +#endif + #ifdef CONFIG_MIGRATION void putback_movable_pages(struct list_head *l); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 10a590ee1c89..50bc43ab50d6 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -1182,6 +1182,7 @@ static struct folio *alloc_migration_target_by_mpol(struct folio *src, h = folio_hstate(src); gfp = htlb_alloc_mask(h); + gfp |= arch_migration_target_gfp(src, gfp); nodemask = policy_nodemask(gfp, pol, ilx, &nid); return alloc_hugetlb_folio_nodemask(h, nid, nodemask, gfp); } @@ -1190,6 +1191,7 @@ static struct folio *alloc_migration_target_by_mpol(struct folio *src, gfp = GFP_TRANSHUGE; else gfp = GFP_HIGHUSER_MOVABLE | __GFP_RETRY_MAYFAIL | __GFP_COMP; + gfp |= arch_migration_target_gfp(src, gfp); page = alloc_pages_mpol(gfp, order, pol, ilx, nid); return page_rmappable_folio(page); diff --git a/mm/migrate.c b/mm/migrate.c index 35a88334bb3c..dd25ab69e3de 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2016,6 +2016,7 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private) struct hstate *h = folio_hstate(src); gfp_mask = htlb_modify_alloc_mask(h, gfp_mask); + gfp_mask |= arch_migration_target_gfp(src, gfp); return alloc_hugetlb_folio_nodemask(h, nid, mtc->nmask, gfp_mask); } @@ -2032,6 +2033,7 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private) zidx = zone_idx(folio_zone(src)); if (is_highmem_idx(zidx) || zidx == ZONE_MOVABLE) gfp_mask |= __GFP_HIGHMEM; + gfp_mask |= arch_migration_target_gfp(src, gfp); return __folio_alloc(gfp_mask, order, nid, mtc->nmask); } @@ -2500,6 +2502,7 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src, __GFP_NOWARN; gfp &= ~__GFP_RECLAIM; } + gfp |= arch_migration_target_gfp(src, gfp); return __folio_alloc_node(gfp, order, nid); } From patchwork Sun Nov 19 16:56:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460592 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 407FEC54E76 for ; Sun, 19 Nov 2023 16:58:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eYzrBZfRTKKptyO3dKLdtURWoPyO5XV9YJoEUANJ6s8=; b=KBLaKLGlXE19vh L5E1v8wN6WfTrue6fbpKfCXvJHAKJ0n1qGJAV0Umr1uHBj8+/YDHZAf15UOyzrM77vDCV6EVYhEeU ppA2gEObgBRldpNxlIOTbinkmvIdsHGFYdg4WwWxnq3aDR8I2NVXNfP8S1tvnZJ3jLD0vyAqJRkg5 5wz8DN9B8zEvz4dfttUzUVh8cbhrax6aJG5+FDxGSw09mRdnmSeIz+OmMVMLKcbQFFgenzVmLdzwC jGvrEw4uLDEO0yLEIHxxG7Tb7W7HZr0s66/Gj4yExtnYKuKhlXPLcWmjtgQqezJTvImMcxrjBAqUH Ac8Rox39lKE1q+lWPcvQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7b-00AQUn-2P; Sun, 19 Nov 2023 16:58:11 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7W-00AQQr-2C for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:08 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 241511477; Sun, 19 Nov 2023 08:58:52 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DB7AF3F6C4; Sun, 19 Nov 2023 08:58:00 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 05/27] mm: page_alloc: Add an arch hook to allow prep_new_page() to fail Date: Sun, 19 Nov 2023 16:56:59 +0000 Message-Id: <20231119165721.9849-6-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085806_848104_8849854F X-CRM114-Status: GOOD ( 18.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Introduce arch_prep_new_page(), which will be used by arm64 to reserve tag storage for an allocated page. Reserving tag storage can fail, for example, if the tag storage page has a short pin on it, so allow prep_new_page() -> arch_prep_new_page() to similarly fail. arch_alloc_page(), called from post_alloc_hook(), has been considered as an alternative to adding yet another arch hook, but post_alloc_hook() cannot fail, as it's also called when free pages are isolated. Signed-off-by: Alexandru Elisei --- include/linux/pgtable.h | 7 ++++ mm/page_alloc.c | 75 ++++++++++++++++++++++++++++++++--------- 2 files changed, 66 insertions(+), 16 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index af7639c3b0a3..b31f53e9ab1d 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -873,6 +873,13 @@ static inline void arch_do_swap_page(struct mm_struct *mm, } #endif +#ifndef __HAVE_ARCH_PREP_NEW_PAGE +static inline int arch_prep_new_page(struct page *page, int order, gfp_t gfp) +{ + return 0; +} +#endif + #ifndef __HAVE_ARCH_UNMAP_ONE /* * Some architectures support metadata associated with a page. When a diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 770e585b77c8..b2782b778e78 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1538,9 +1538,15 @@ inline void post_alloc_hook(struct page *page, unsigned int order, page_table_check_alloc(page, order); } -static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, +static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags) { + int ret; + + ret = arch_prep_new_page(page, order, gfp_flags); + if (unlikely(ret)) + return ret; + post_alloc_hook(page, order, gfp_flags); if (order && (gfp_flags & __GFP_COMP)) @@ -1556,6 +1562,8 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags set_page_pfmemalloc(page); else clear_page_pfmemalloc(page); + + return 0; } /* @@ -3163,6 +3171,24 @@ static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask, return alloc_flags; } +#ifdef HAVE_ARCH_ALLOC_PAGE +static void return_page_to_buddy(struct page *page, int order) +{ + int migratetype = get_pfnblock_migratetype(page, pfn); + unsigned long pfn = page_to_pfn(page); + struct zone *zone = page_zone(page); + unsigned long flags; + + spin_lock_irqsave(&zone->lock, flags); + __free_one_page(page, pfn, zone, order, migratetype, FPI_TO_TAIL); + spin_unlock_irqrestore(&zone->lock, flags); +} +#else +static void return_page_to_buddy(struct page *page, int order) +{ +} +#endif + /* * get_page_from_freelist goes through the zonelist trying to allocate * a page. @@ -3309,7 +3335,10 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, page = rmqueue(ac->preferred_zoneref->zone, zone, order, gfp_mask, alloc_flags, ac->migratetype); if (page) { - prep_new_page(page, order, gfp_mask, alloc_flags); + if (prep_new_page(page, order, gfp_mask, alloc_flags)) { + return_page_to_buddy(page, order); + goto no_page; + } /* * If this is a high-order atomic allocation then check @@ -3319,20 +3348,20 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, reserve_highatomic_pageblock(page, zone); return page; - } else { - if (has_unaccepted_memory()) { - if (try_to_accept_memory(zone, order)) - goto try_this_zone; - } + } +no_page: + if (has_unaccepted_memory()) { + if (try_to_accept_memory(zone, order)) + goto try_this_zone; + } #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT - /* Try again if zone has deferred pages */ - if (deferred_pages_enabled()) { - if (_deferred_grow_zone(zone, order)) - goto try_this_zone; - } -#endif + /* Try again if zone has deferred pages */ + if (deferred_pages_enabled()) { + if (_deferred_grow_zone(zone, order)) + goto try_this_zone; } +#endif } /* @@ -3538,8 +3567,12 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order, count_vm_event(COMPACTSTALL); /* Prep a captured page if available */ - if (page) - prep_new_page(page, order, gfp_mask, alloc_flags); + if (page) { + if (prep_new_page(page, order, gfp_mask, alloc_flags)) { + return_page_to_buddy(page, order); + page = NULL; + } + } /* Try get a page from the freelist if available */ if (!page) @@ -4490,9 +4523,18 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, } break; } + + if (prep_new_page(page, 0, gfp, 0)) { + pcp_spin_unlock(pcp); + pcp_trylock_finish(UP_flags); + return_page_to_buddy(page, 0); + if (!nr_account) + goto failed; + else + goto out_statistics; + } nr_account++; - prep_new_page(page, 0, gfp, 0); if (page_list) list_add(&page->lru, page_list); else @@ -4503,6 +4545,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, pcp_spin_unlock(pcp); pcp_trylock_finish(UP_flags); +out_statistics: __count_zid_vm_events(PGALLOC, zone_idx(zone), nr_account); zone_statistics(ac.preferred_zoneref->zone, zone, nr_account); From patchwork Sun Nov 19 16:57:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460593 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA365C072A2 for ; Sun, 19 Nov 2023 16:58:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=6zmOBVZMV6AsWBR70QnWa6yIP9fxevPaSRx4rJeoc4k=; b=m2a5MT9CkgT65E 2422D4Uzdh4okkf1Hk4po012hiw6YpL9GSpUUUuxy8k9o6jOOMNcU0wJ+QWnE8lwCeddUI/eRnXFR zJBLGHztBnrc3N0iAG1Y2Xk//W8BqK/JxSklp3B3kabRBsvYmir8DB0zJjUjpYNd3rUmskesAj+Qp GCK1pJFswQBjH8/x0vHVcnT6ckNcG1lR1Vx4SibUpB2zR6TuGGSy1utukB2wvTR+IVB9BdIlhyJZs zoXFgnoYZH9xtyI4lNWmVJ20UdIvYPxkFcVikujp0gvGWroFBcpHJVre4SnG4mpzcTn4H0u5JE2Ig UuZlZDTGuXh0j3sHLNSw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7l-00AQcQ-1M; Sun, 19 Nov 2023 16:58:21 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7b-00AQUi-2E for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:13 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 68FED1480; Sun, 19 Nov 2023 08:58:57 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2ED763F6C4; Sun, 19 Nov 2023 08:58:06 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 06/27] mm: page_alloc: Allow an arch to hook early into free_pages_prepare() Date: Sun, 19 Nov 2023 16:57:00 +0000 Message-Id: <20231119165721.9849-7-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085811_783919_D3E30D46 X-CRM114-Status: GOOD ( 11.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add arch_free_pages_prepare() hook that is called before that page flags are cleared. This will be used by arm64 when explicit management of tag storage pages is enabled. Signed-off-by: Alexandru Elisei --- include/linux/pgtable.h | 4 ++++ mm/page_alloc.c | 4 +++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index b31f53e9ab1d..3f34f00ced62 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -880,6 +880,10 @@ static inline int arch_prep_new_page(struct page *page, int order, gfp_t gfp) } #endif +#ifndef __HAVE_ARCH_FREE_PAGES_PREPARE +static inline void arch_free_pages_prepare(struct page *page, int order) { } +#endif + #ifndef __HAVE_ARCH_UNMAP_ONE /* * Some architectures support metadata associated with a page. When a diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b2782b778e78..86e4b1dac538 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1086,6 +1086,8 @@ static __always_inline bool free_pages_prepare(struct page *page, trace_mm_page_free(page, order); kmsan_free_page(page, order); + arch_free_pages_prepare(page, order); + if (unlikely(PageHWPoison(page)) && !order) { /* * Do not let hwpoison pages hit pcplists/buddy @@ -3171,7 +3173,7 @@ static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask, return alloc_flags; } -#ifdef HAVE_ARCH_ALLOC_PAGE +#ifdef HAVE_ARCH_PREP_NEW_PAGE static void return_page_to_buddy(struct page *page, int order) { int migratetype = get_pfnblock_migratetype(page, pfn); From patchwork Sun Nov 19 16:57:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460594 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6FA97C072A2 for ; Sun, 19 Nov 2023 16:58:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=09fHrmOtmDfr3Lkdi7WvV7FSqzukdIR8L3OZtsjQXRA=; b=j44iWSRhAsrStm MdHBPZcXoxsNvBYmRRlTZ726MoG2PEQJvUn+n5OB/tFWHsRpz+n2BYkqfWL6lt8l8T9U7D7JpTLOW ZnnOulvuq6nwPiChkc6vdi9QGMfh4lVFE8vxX7lgPH3YJ2lpSLnZWoyJKAeR5pV+za6mGE8NQdE5m UQl7jX9Eqq6bsY0CYd/pmdlHxBA61aa6QOVh0F6G7kVxnqtRjlHaoYHNsRjFnqnjec+GVoAhJW6QZ MO11uAy501e4NgkJo6OvqttMyFKECZdFKewTn5y+ozMF5wmhDHluCvLR1kA66s+OoO5vZF2gFdx5D EvgSnwsFrDwYHIWtphKw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7q-00AQhG-20; Sun, 19 Nov 2023 16:58:26 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7i-00AQa7-0L for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:20 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 967AFDA7; Sun, 19 Nov 2023 08:59:02 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6F4F03F6C4; Sun, 19 Nov 2023 08:58:11 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 07/27] mm: page_alloc: Add an arch hook to filter MIGRATE_CMA allocations Date: Sun, 19 Nov 2023 16:57:01 +0000 Message-Id: <20231119165721.9849-8-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085818_246409_6BADF164 X-CRM114-Status: GOOD ( 10.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As an architecture might have specific requirements around the allocation of CMA pages, add an arch hook that can disable allocations from MIGRATE_CMA, if the allocation was otherwise allowed. This will be used by arm64, which will put tag storage pages on the MIGRATE_CMA list, pages which have specific limitations. Signed-off-by: Alexandru Elisei --- include/linux/pgtable.h | 7 +++++++ mm/page_alloc.c | 3 ++- 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 3f34f00ced62..b7a9ab818f6d 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -884,6 +884,13 @@ static inline int arch_prep_new_page(struct page *page, int order, gfp_t gfp) static inline void arch_free_pages_prepare(struct page *page, int order) { } #endif +#ifndef __HAVE_ARCH_ALLOC_CMA +static inline bool arch_alloc_cma(gfp_t gfp) +{ + return true; +} +#endif + #ifndef __HAVE_ARCH_UNMAP_ONE /* * Some architectures support metadata associated with a page. When a diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 86e4b1dac538..0f508070c404 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3167,7 +3167,8 @@ static inline unsigned int gfp_to_alloc_flags_cma(gfp_t gfp_mask, unsigned int alloc_flags) { #ifdef CONFIG_CMA - if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE) + if (gfp_migratetype(gfp_mask) == MIGRATE_MOVABLE && + arch_alloc_cma(gfp_mask)) alloc_flags |= ALLOC_CMA; #endif return alloc_flags; From patchwork Sun Nov 19 16:57:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460596 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 735E0C54FB9 for ; Sun, 19 Nov 2023 16:58:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ZW0WhRk/PncnXrwG6j+EdZU+2HMTX/eH1f5pLAUozQE=; b=oKXALfwr+wiyPV gYvuH982RIVA7ePqmGBJZl+fXG35rs94TJ+U8q4Jrnum/ddYf3aK5KoHj6f84CXqNUit8p5pC4/9I Un/A1yAnGl4D2raFmjZhQpHfsay8OwpDc43KAqUb0cEqQcRE2SImYliQXYvfwx+srYjzAPVnwITem ya+BV1JmxQnFVS0q7CjNWkRWGJgtmhNc3QzOT+nw3RQVpYLbbfM9WYtQPtAUyxewpCHaiQbrcf79e ytv6RH4lJnLpvc9ud/DJ+O5zkqRLVkj2/ENFmXS6UzuVmF+EDcjHeYTJnZsEWtTuj7+cTPnQbXNj0 k7GveiwsiEXRGUlg4rgw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7r-00AQiG-1L; Sun, 19 Nov 2023 16:58:27 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7m-00AQd1-0v for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:23 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DABF9FEC; Sun, 19 Nov 2023 08:59:07 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A20A03F6C4; Sun, 19 Nov 2023 08:58:16 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 08/27] mm: page_alloc: Partially revert "mm: page_alloc: remove stale CMA guard code" Date: Sun, 19 Nov 2023 16:57:02 +0000 Message-Id: <20231119165721.9849-9-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085822_433318_3929C624 X-CRM114-Status: GOOD ( 11.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The patch f945116e4e19 ("mm: page_alloc: remove stale CMA guard code") removed the CMA filter when allocating from the MIGRATE_MOVABLE pcp list because CMA is always allowed when __GFP_MOVABLE is set. With the introduction of the arch_alloc_cma() function, the above is not true anymore, so bring back the filter. This is a partially revert because the stale comment remains removed. Signed-off-by: Alexandru Elisei --- mm/page_alloc.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0f508070c404..135f9283a863 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2907,10 +2907,17 @@ struct page *rmqueue(struct zone *preferred_zone, WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1)); if (likely(pcp_allowed_order(order))) { - page = rmqueue_pcplist(preferred_zone, zone, order, - migratetype, alloc_flags); - if (likely(page)) - goto out; + /* + * MIGRATE_MOVABLE pcplist could have the pages on CMA area and + * we need to skip it when CMA area isn't allowed. + */ + if (!IS_ENABLED(CONFIG_CMA) || alloc_flags & ALLOC_CMA || + migratetype != MIGRATE_MOVABLE) { + page = rmqueue_pcplist(preferred_zone, zone, order, + migratetype, alloc_flags); + if (likely(page)) + goto out; + } } page = rmqueue_buddy(preferred_zone, zone, order, alloc_flags, From patchwork Sun Nov 19 16:57:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DD92CC072A2 for ; Sun, 19 Nov 2023 16:58:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GGK5+EeYYKXLtd/RM2hL+P6D7meiOL8EFYngy9ufOPs=; b=F9eR7lYwQffUiC lsOwieNkOn3Icu5+w/3LCHd4aKaXVSDhEt0kUYr6Cb46c01168woNqBqTQ0s0d0O3noCp3Obqdi6M jXS6tAFCjnOBvjz0jAuIaug6dgubEfnaFtBHRRGnxfc9FDhv5JvU49p5lgQyy4o3hiI79Aw3Eyf1f xZd61TzVzhhQASgEirMl5/paDsuXZm6jXMSC6TW9PdAlElAMBWSy9RC3RNokULMMY2XSBJiyT/7CC AYeXLG3S6i14MVLW+2WWaPGGmpX9VmSlM0ClUhh/DFuNSrkP+8bhZoqUEIHyeLclX4m4bV2AsQUA8 dbLTJBIXQBK9d1n/u0PQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7y-00AQoy-2N; Sun, 19 Nov 2023 16:58:34 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7r-00AQiE-1b for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:29 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2C9E914BF; Sun, 19 Nov 2023 08:59:13 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E6E3F3F6C4; Sun, 19 Nov 2023 08:58:21 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 09/27] mm: Allow an arch to hook into folio allocation when VMA is known Date: Sun, 19 Nov 2023 16:57:03 +0000 Message-Id: <20231119165721.9849-10-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085827_830329_0FBB8369 X-CRM114-Status: GOOD ( 20.00 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org An architecture might want to fixup the gfp flags based on the type of VMA where the page will be mapped. On arm64, this is currently used if the VMA is MTE enabled. When __GFP_TAGGED is set, for performance reasons, tag zeroing is performed at the same time as the data is zeroed, instead of being performed separately, in set_pte_at() -> mte_sync_tags(). Its usage will be expanded when the storage for the tags will have to be explicitely managed by the kernel. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/page.h | 5 ++--- arch/arm64/include/asm/pgtable.h | 3 +++ arch/arm64/mm/fault.c | 19 ++++++------------- include/linux/pgtable.h | 7 +++++++ mm/mempolicy.c | 1 + mm/shmem.c | 5 ++++- 6 files changed, 23 insertions(+), 17 deletions(-) diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h index 2312e6ee595f..c8125a28eaa2 100644 --- a/arch/arm64/include/asm/page.h +++ b/arch/arm64/include/asm/page.h @@ -29,9 +29,8 @@ void copy_user_highpage(struct page *to, struct page *from, void copy_highpage(struct page *to, struct page *from); #define __HAVE_ARCH_COPY_HIGHPAGE -struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr); -#define vma_alloc_zeroed_movable_folio vma_alloc_zeroed_movable_folio +#define vma_alloc_zeroed_movable_folio(vma, vaddr) \ + vma_alloc_folio(GFP_HIGHUSER_MOVABLE | __GFP_ZERO, 0, vma, vaddr, false); void tag_clear_highpage(struct page *to); #define __HAVE_ARCH_TAG_CLEAR_HIGHPAGE diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9b32c74b4a1b..cd5dacd1be3a 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1065,6 +1065,9 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) #endif /* CONFIG_ARM64_MTE */ +#define __HAVE_ARCH_CALC_VMA_GFP +gfp_t arch_calc_vma_gfp(struct vm_area_struct *vma, gfp_t gfp); + /* * On AArch64, the cache coherency is handled via the set_pte_at() function. */ diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index daa91608d917..acbc7530d2b2 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -935,22 +935,15 @@ void do_debug_exception(unsigned long addr_if_watchpoint, unsigned long esr, NOKPROBE_SYMBOL(do_debug_exception); /* - * Used during anonymous page fault handling. + * If this is called during anonymous page fault handling, and the page is + * mapped with PROT_MTE, initialise the tags at the point of tag zeroing as this + * is usually faster than separate DC ZVA and STGM. */ -struct folio *vma_alloc_zeroed_movable_folio(struct vm_area_struct *vma, - unsigned long vaddr) +gfp_t arch_calc_vma_gfp(struct vm_area_struct *vma, gfp_t gfp) { - gfp_t flags = GFP_HIGHUSER_MOVABLE | __GFP_ZERO; - - /* - * If the page is mapped with PROT_MTE, initialise the tags at the - * point of allocation and page zeroing as this is usually faster than - * separate DC ZVA and STGM. - */ if (vma->vm_flags & VM_MTE) - flags |= __GFP_TAGGED; - - return vma_alloc_folio(flags, 0, vma, vaddr, false); + return __GFP_TAGGED; + return 0; } void tag_clear_highpage(struct page *page) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index b7a9ab818f6d..b1001ce361ac 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -873,6 +873,13 @@ static inline void arch_do_swap_page(struct mm_struct *mm, } #endif +#ifndef __HAVE_ARCH_CALC_VMA_GFP +static inline gfp_t arch_calc_vma_gfp(struct vm_area_struct *vma, gfp_t gfp) +{ + return 0; +} +#endif + #ifndef __HAVE_ARCH_PREP_NEW_PAGE static inline int arch_prep_new_page(struct page *page, int order, gfp_t gfp) { diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 50bc43ab50d6..cb170abae1fd 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2170,6 +2170,7 @@ struct folio *vma_alloc_folio(gfp_t gfp, int order, struct vm_area_struct *vma, pgoff_t ilx; struct page *page; + gfp |= arch_calc_vma_gfp(vma, gfp); pol = get_vma_policy(vma, addr, order, &ilx); page = alloc_pages_mpol(gfp | __GFP_COMP, order, pol, ilx, numa_node_id()); diff --git a/mm/shmem.c b/mm/shmem.c index 91e2620148b2..71ce5fe5c779 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1570,7 +1570,7 @@ static struct folio *shmem_swapin_cluster(swp_entry_t swap, gfp_t gfp, */ static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp) { - gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM; + gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM | __GFP_TAGGED; gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY; gfp_t zoneflags = limit_gfp & GFP_ZONEMASK; gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK); @@ -2023,6 +2023,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, gfp_t huge_gfp; huge_gfp = vma_thp_gfp_mask(vma); + huge_gfp |= arch_calc_vma_gfp(vma, huge_gfp); huge_gfp = limit_gfp_mask(huge_gfp, gfp); folio = shmem_alloc_and_add_folio(huge_gfp, inode, index, fault_mm, true); @@ -2199,6 +2200,8 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) vm_fault_t ret = 0; int err; + gfp |= arch_calc_vma_gfp(vmf->vma, gfp); + /* * Trinity finds that probing a hole which tmpfs is punching can * prevent the hole-punch from ever completing: noted in i_private. From patchwork Sun Nov 19 16:57:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 481D5C54E76 for ; Sun, 19 Nov 2023 16:59:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ljTngy15Yoi3DQMWfk10M5eF3RUkeEXA4jDrdS05YN4=; b=FnsyQYWbUWMgcL 14ncnc5TlrEv5f3+bqS3l5dUeoEuI91J9orQkzAfih9Hou54XagYrkrigSqoCsuVdK2iRc3B/dFl3 W3abIXS2aanTECCAyc8gnoIjpS4B0tJlqLVwRtOJAuuCcC9c9k4apHiEeCOhaSiEfbd86HeqMp0rH lkXIgYq3vOnivcfLAu1C+wdrslDiARgtD7p8r96kQ3gw2vBQFJHoQwkHqwO1WVQlKyJCuP9xPDowp ZKqi4jygm+paGkzWk+BrppUB0HsSiN+aSKxP+YRauEDvqLcbyDgIhL8C/yL8V5oJS/kWLi8RKfqsF VnNrBy2660xRqSG/Bdjw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l83-00AQt3-2z; Sun, 19 Nov 2023 16:58:39 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l7w-00AQnT-2W for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:34 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 87A641042; Sun, 19 Nov 2023 08:59:18 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 40DE23F6C4; Sun, 19 Nov 2023 08:58:27 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 10/27] mm: Call arch_swap_prepare_to_restore() before arch_swap_restore() Date: Sun, 19 Nov 2023 16:57:04 +0000 Message-Id: <20231119165721.9849-11-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085832_990225_9AC267D1 X-CRM114-Status: GOOD ( 12.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org arm64 uses arch_swap_restore() to restore saved tags before the page is swapped in and it's called in atomic context (with the ptl lock held). Introduce arch_swap_prepare_to_restore() that will allow an architecture to perform extra work during swap in and outside of a critical section. Signed-off-by: Alexandru Elisei --- include/linux/pgtable.h | 7 +++++++ mm/memory.c | 4 ++++ mm/shmem.c | 9 +++++++++ mm/swapfile.c | 7 +++++++ 4 files changed, 27 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index b1001ce361ac..ffdb9b6bed6c 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -938,6 +938,13 @@ static inline void arch_swap_invalidate_area(int type) } #endif +#ifndef __HAVE_ARCH_SWAP_PREPARE_TO_RESTORE +static inline vm_fault_t arch_swap_prepare_to_restore(swp_entry_t entry, struct folio *folio) +{ + return 0; +} +#endif + #ifndef __HAVE_ARCH_SWAP_RESTORE static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) { diff --git a/mm/memory.c b/mm/memory.c index 1f18ed4a5497..e137f7673749 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3957,6 +3957,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) folio_throttle_swaprate(folio, GFP_KERNEL); + ret = arch_swap_prepare_to_restore(entry, folio); + if (ret) + goto out_page; + /* * Back out if somebody else already faulted in this pte. */ diff --git a/mm/shmem.c b/mm/shmem.c index 71ce5fe5c779..0449c03dbdfd 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1840,6 +1840,7 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, struct swap_info_struct *si; struct folio *folio = NULL; swp_entry_t swap; + vm_fault_t ret; int error; VM_BUG_ON(!*foliop || !xa_is_value(*foliop)); @@ -1888,6 +1889,14 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index, } folio_wait_writeback(folio); + ret = arch_swap_prepare_to_restore(swap, folio); + if (ret) { + if (fault_type) + *fault_type = ret; + error = -EINVAL; + goto unlock; + } + /* * Some architectures may have to restore extra metadata to the * folio after reading from swap. diff --git a/mm/swapfile.c b/mm/swapfile.c index 4bc70f459164..9983dffce47b 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1746,6 +1746,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, spinlock_t *ptl; pte_t *pte, new_pte, old_pte; bool hwpoisoned = PageHWPoison(page); + vm_fault_t err; int ret = 1; swapcache = page; @@ -1779,6 +1780,12 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, goto setpte; } + err = arch_swap_prepare_to_restore(entry, page_folio(page)); + if (err) { + ret = -EINVAL; + goto out; + } + /* * Some architectures may have to restore extra metadata to the page * when reading from swap. This metadata may be indexed by swap entry From patchwork Sun Nov 19 16:57:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460598 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 69D78C54E76 for ; Sun, 19 Nov 2023 16:59:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=P5NSQdr9d4xAS3DvueLoEteDtrjjSq3ti+Sho+9Osh0=; b=EgmtrSmByYxPUz g3V9ATh3AUQ79F0XPejyq8mPgIMQjkiw56xI1OBjuM9U8S3E2Dyiz9RMGDzCMfAqci2CQIkLGTa83 PVOCtyYIO6zwd0mJhXpNi1jqa1TNhg8FTlOpCgS5HHCFZlhcEjz5L1Y3l3Jweqsp1ZpvvEoMCC66D 7IFxoFQYJLvi91OJccF/Sx4d2y7dXwmvSIwaaccENSI+FqMNSS00aXRSEdGkC8iuBQHygA7do6sAw 9Jvx6dhcWA7a74iGcq/6QZKr2i9Pu78Qxe+/8CbPscKUIZ9SIPF433WndXMxy3iYUa8kQ8DmLiWXp FNEU+HDIKdFMvzqBu+iw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l89-00AQxp-1A; Sun, 19 Nov 2023 16:58:45 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l82-00AQrR-06 for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:40 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BD2561007; Sun, 19 Nov 2023 08:59:23 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 92A023F6C4; Sun, 19 Nov 2023 08:58:32 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 11/27] arm64: mte: Reserve tag storage memory Date: Sun, 19 Nov 2023 16:57:05 +0000 Message-Id: <20231119165721.9849-12-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085838_177243_66C3BF05 X-CRM114-Status: GOOD ( 30.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Allow the kernel to get the size and location of the MTE tag storage regions from the DTB. This memory is marked as reserved for now. The DTB node for the tag storage region is defined as: tags0: tag-storage@8f8000000 { compatible = "arm,mte-tag-storage"; reg = <0x08 0xf8000000 0x00 0x4000000>; block-size = <0x1000>; memory = <&memory0>; // Associated tagged memory node }; The tag storage region represents the largest contiguous memory region that holds all the tags for the associated contiguous memory region which can be tagged. For example, for a 32GB contiguous tagged memory the corresponding tag storage region is 1GB of contiguous memory, not two adjacent 512M of tag storage memory. "block-size" represents the minimum multiple of 4K of tag storage where all the tags stored in the block correspond to a contiguous memory region. This is needed for platforms where the memory controller interleaves tag writes to memory. For example, if the memory controller interleaves tag writes for 256KB of contiguous memory across 8K of tag storage (2-way interleave), then the correct value for "block-size" is 0x2000. This value is a hardware property, independent of the selected kernel page size. Signed-off-by: Alexandru Elisei --- arch/arm64/Kconfig | 12 ++ arch/arm64/include/asm/mte_tag_storage.h | 15 ++ arch/arm64/kernel/Makefile | 1 + arch/arm64/kernel/mte_tag_storage.c | 256 +++++++++++++++++++++++ arch/arm64/kernel/setup.c | 7 + 5 files changed, 291 insertions(+) create mode 100644 arch/arm64/include/asm/mte_tag_storage.h create mode 100644 arch/arm64/kernel/mte_tag_storage.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 7b071a00425d..fe8276fdc7a8 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2062,6 +2062,18 @@ config ARM64_MTE Documentation/arch/arm64/memory-tagging-extension.rst. +if ARM64_MTE +config ARM64_MTE_TAG_STORAGE + bool "Dynamic MTE tag storage management" + help + Adds support for dynamic management of the memory used by the hardware + for storing MTE tags. This memory, unlike normal memory, cannot be + tagged. When it is used to store tags for another memory location it + cannot be used for any type of allocation. + + If unsure, say N +endif # ARM64_MTE + endmenu # "ARMv8.5 architectural features" menu "ARMv8.7 architectural features" diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h new file mode 100644 index 000000000000..8f86c4f9a7c3 --- /dev/null +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -0,0 +1,15 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (C) 2023 ARM Ltd. + */ +#ifndef __ASM_MTE_TAG_STORAGE_H +#define __ASM_MTE_TAG_STORAGE_H + +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +void mte_tag_storage_init(void); +#else +static inline void mte_tag_storage_init(void) +{ +} +#endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ +#endif /* __ASM_MTE_TAG_STORAGE_H */ diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index d95b3d6b471a..5f031bf9f8f1 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -70,6 +70,7 @@ obj-$(CONFIG_CRASH_CORE) += crash_core.o obj-$(CONFIG_ARM_SDE_INTERFACE) += sdei.o obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o +obj-$(CONFIG_ARM64_MTE_TAG_STORAGE) += mte_tag_storage.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c new file mode 100644 index 000000000000..fa6267ef8392 --- /dev/null +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -0,0 +1,256 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Support for dynamic tag storage. + * + * Copyright (C) 2023 ARM Ltd. + */ + +#include +#include +#include +#include +#include +#include +#include + +#include + +struct tag_region { + struct range mem_range; /* Memory associated with the tag storage, in PFNs. */ + struct range tag_range; /* Tag storage memory, in PFNs. */ + u32 block_size; /* Tag block size, in pages. */ +}; + +#define MAX_TAG_REGIONS 32 + +static struct tag_region tag_regions[MAX_TAG_REGIONS]; +static int num_tag_regions; + +static int __init tag_storage_of_flat_get_range(unsigned long node, const __be32 *reg, + int reg_len, struct range *range) +{ + int addr_cells = dt_root_addr_cells; + int size_cells = dt_root_size_cells; + u64 size; + + if (reg_len / 4 > addr_cells + size_cells) + return -EINVAL; + + range->start = PHYS_PFN(of_read_number(reg, addr_cells)); + size = PHYS_PFN(of_read_number(reg + addr_cells, size_cells)); + if (size == 0) { + pr_err("Invalid node"); + return -EINVAL; + } + range->end = range->start + size - 1; + + return 0; +} + +static int __init tag_storage_of_flat_get_tag_range(unsigned long node, + struct range *tag_range) +{ + const __be32 *reg; + int reg_len; + + reg = of_get_flat_dt_prop(node, "reg", ®_len); + if (reg == NULL) { + pr_err("Invalid metadata node"); + return -EINVAL; + } + + return tag_storage_of_flat_get_range(node, reg, reg_len, tag_range); +} + +static int __init tag_storage_of_flat_get_memory_range(unsigned long node, struct range *mem) +{ + const __be32 *reg; + int reg_len; + + reg = of_get_flat_dt_prop(node, "linux,usable-memory", ®_len); + if (reg == NULL) + reg = of_get_flat_dt_prop(node, "reg", ®_len); + + if (reg == NULL) { + pr_err("Invalid memory node"); + return -EINVAL; + } + + return tag_storage_of_flat_get_range(node, reg, reg_len, mem); +} + +struct find_memory_node_arg { + unsigned long node; + u32 phandle; +}; + +static int __init fdt_find_memory_node(unsigned long node, const char *uname, + int depth, void *data) +{ + const char *type = of_get_flat_dt_prop(node, "device_type", NULL); + struct find_memory_node_arg *arg = data; + + if (depth != 1 || !type || strcmp(type, "memory") != 0) + return 0; + + if (of_get_flat_dt_phandle(node) == arg->phandle) { + arg->node = node; + return 1; + } + + return 0; +} + +static int __init tag_storage_get_memory_node(unsigned long tag_node, unsigned long *mem_node) +{ + struct find_memory_node_arg arg = { 0 }; + const __be32 *memory_prop; + u32 mem_phandle; + int ret, reg_len; + + memory_prop = of_get_flat_dt_prop(tag_node, "memory", ®_len); + if (!memory_prop) { + pr_err("Missing 'memory' property in the tag storage node"); + return -EINVAL; + } + + mem_phandle = be32_to_cpup(memory_prop); + arg.phandle = mem_phandle; + + ret = of_scan_flat_dt(fdt_find_memory_node, &arg); + if (ret != 1) { + pr_err("Associated memory node not found"); + return -EINVAL; + } + + *mem_node = arg.node; + + return 0; +} + +static int __init tag_storage_of_flat_read_u32(unsigned long node, const char *propname, + u32 *retval) +{ + const __be32 *reg; + + reg = of_get_flat_dt_prop(node, propname, NULL); + if (!reg) + return -EINVAL; + + *retval = be32_to_cpup(reg); + return 0; +} + +static u32 __init get_block_size_pages(u32 block_size_bytes) +{ + u32 a = PAGE_SIZE; + u32 b = block_size_bytes; + u32 r; + + /* Find greatest common divisor using the Euclidian algorithm. */ + do { + r = a % b; + a = b; + b = r; + } while (b != 0); + + return PHYS_PFN(PAGE_SIZE * block_size_bytes / a); +} + +static int __init fdt_init_tag_storage(unsigned long node, const char *uname, + int depth, void *data) +{ + struct tag_region *region; + unsigned long mem_node; + struct range *mem_range; + struct range *tag_range; + u32 block_size_bytes; + u32 nid = 0; + int ret; + + if (depth != 1 || !strstr(uname, "tag-storage")) + return 0; + + if (!of_flat_dt_is_compatible(node, "arm,mte-tag-storage")) + return 0; + + if (num_tag_regions == MAX_TAG_REGIONS) { + pr_err("Maximum number of tag storage regions exceeded"); + return -EINVAL; + } + + region = &tag_regions[num_tag_regions]; + mem_range = ®ion->mem_range; + tag_range = ®ion->tag_range; + + ret = tag_storage_of_flat_get_tag_range(node, tag_range); + if (ret) { + pr_err("Invalid tag storage node"); + return ret; + } + + ret = tag_storage_get_memory_node(node, &mem_node); + if (ret) + return ret; + + ret = tag_storage_of_flat_get_memory_range(mem_node, mem_range); + if (ret) { + pr_err("Invalid address for associated data memory node"); + return ret; + } + + /* The tag region must exactly match the corresponding memory. */ + if (range_len(tag_range) * 32 != range_len(mem_range)) { + pr_err("Tag storage region 0x%llx-0x%llx does not cover the memory region 0x%llx-0x%llx", + PFN_PHYS(tag_range->start), PFN_PHYS(tag_range->end), + PFN_PHYS(mem_range->start), PFN_PHYS(mem_range->end)); + return -EINVAL; + } + + ret = tag_storage_of_flat_read_u32(node, "block-size", &block_size_bytes); + if (ret || block_size_bytes == 0) { + pr_err("Invalid or missing 'block-size' property"); + return -EINVAL; + } + region->block_size = get_block_size_pages(block_size_bytes); + if (range_len(tag_range) % region->block_size != 0) { + pr_err("Tag storage region size 0x%llx is not a multiple of block size %u", + PFN_PHYS(range_len(tag_range)), region->block_size); + return -EINVAL; + } + + ret = tag_storage_of_flat_read_u32(mem_node, "numa-node-id", &nid); + if (ret) + nid = numa_node_id(); + + ret = memblock_add_node(PFN_PHYS(tag_range->start), PFN_PHYS(range_len(tag_range)), + nid, MEMBLOCK_NONE); + if (ret) { + pr_err("Error adding tag memblock (%d)", ret); + return ret; + } + memblock_reserve(PFN_PHYS(tag_range->start), PFN_PHYS(range_len(tag_range))); + + pr_info("Found tag storage region 0x%llx-0x%llx, block size %u pages", + PFN_PHYS(tag_range->start), PFN_PHYS(tag_range->end), region->block_size); + + num_tag_regions++; + + return 0; +} + +void __init mte_tag_storage_init(void) +{ + struct range *tag_range; + int i, ret; + + ret = of_scan_flat_dt(fdt_init_tag_storage, NULL); + if (ret) { + for (i = 0; i < num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + memblock_remove(PFN_PHYS(tag_range->start), PFN_PHYS(range_len(tag_range))); + } + num_tag_regions = 0; + pr_info("MTE tag storage region management disabled"); + } +} diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index 417a8a86b2db..1b77138c1aa5 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -342,6 +343,12 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) FW_BUG "Booted with MMU enabled!"); } + /* + * Must be called before memory limits are enforced by + * arm64_memblock_init(). + */ + mte_tag_storage_init(); + arm64_memblock_init(); paging_init(); From patchwork Sun Nov 19 16:57:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5FB72C5AE5B for ; Sun, 19 Nov 2023 16:59:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=67OmKL/BO3LqBpYuTT2fRkGjT2LCGJvWxW1d+ptDOwA=; b=yJwrdRO4GxCy7I OP/Z3dGcKy/vyD6WJQesUVePyVNyMRSiLyOMZE3ESu9kVLoJQ+OmPMXACHVodCPCfmLtZhAP4dmt0 14i63mljh1s7wXl/UQgBVbPO4Y9bkahd0QXf4LNWI3bFi62ikcjfne23PELFY+DzaLVb4uujtEwKM ZmwE0U/+QQ+xlfFbCex6cF1RVyRY1Fvi+onEmGs8o9fJWkOzDdyvU6woJeUtqSJw7klFoc/oFIuwm g1MpM7MNEX9UWY27QFmcVe0s31pk+2TcGZnJwCrPjIeoe8gpqydZh9f4uECRc09QsXes6DWSUWVZr 1NrukCb6Tbaog1b+05zQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8F-00AR3M-1N; Sun, 19 Nov 2023 16:58:51 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l87-00AQvP-0w for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:46 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1B42E1476; Sun, 19 Nov 2023 08:59:29 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C846B3F6C4; Sun, 19 Nov 2023 08:58:37 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 12/27] arm64: mte: Add tag storage pages to the MIGRATE_CMA migratetype Date: Sun, 19 Nov 2023 16:57:06 +0000 Message-Id: <20231119165721.9849-13-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085843_461637_2F4E1D97 X-CRM114-Status: GOOD ( 19.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Add the MTE tag storage pages to the MIGRATE_CMA migratetype, which allows the page allocator to manage them like regular pages. Ths migratype lends the pages some very desirable properties: * They cannot be longterm pinned, meaning they will always be migratable. * The pages can be allocated explicitely by using their PFN (with alloc_contig_range()) when they are needed to store tags. Signed-off-by: Alexandru Elisei --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/mte_tag_storage.c | 68 +++++++++++++++++++++++++++++ include/linux/mmzone.h | 5 +++ mm/internal.h | 3 -- 4 files changed, 74 insertions(+), 3 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index fe8276fdc7a8..047487046e8f 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2065,6 +2065,7 @@ config ARM64_MTE if ARM64_MTE config ARM64_MTE_TAG_STORAGE bool "Dynamic MTE tag storage management" + select CONFIG_CMA help Adds support for dynamic management of the memory used by the hardware for storing MTE tags. This memory, unlike normal memory, cannot be diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index fa6267ef8392..427f4f1909f3 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -5,10 +5,12 @@ * Copyright (C) 2023 ARM Ltd. */ +#include #include #include #include #include +#include #include #include #include @@ -189,6 +191,14 @@ static int __init fdt_init_tag_storage(unsigned long node, const char *uname, return ret; } + /* Pages are managed in pageblock_nr_pages chunks */ + if (!IS_ALIGNED(tag_range->start | range_len(tag_range), pageblock_nr_pages)) { + pr_err("Tag storage region 0x%llx-0x%llx not aligned to pageblock size 0x%llx", + PFN_PHYS(tag_range->start), PFN_PHYS(tag_range->end), + PFN_PHYS(pageblock_nr_pages)); + return -EINVAL; + } + ret = tag_storage_get_memory_node(node, &mem_node); if (ret) return ret; @@ -254,3 +264,61 @@ void __init mte_tag_storage_init(void) pr_info("MTE tag storage region management disabled"); } } + +static int __init mte_tag_storage_activate_regions(void) +{ + phys_addr_t dram_start, dram_end; + struct range *tag_range; + unsigned long pfn; + int i, ret; + + if (num_tag_regions == 0) + return 0; + + dram_start = memblock_start_of_DRAM(); + dram_end = memblock_end_of_DRAM(); + + for (i = 0; i < num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + /* + * Tag storage region was clipped by arm64_bootmem_init() + * enforcing addressing limits. + */ + if (PFN_PHYS(tag_range->start) < dram_start || + PFN_PHYS(tag_range->end) >= dram_end) { + pr_err("Tag storage region 0x%llx-0x%llx outside addressable memory", + PFN_PHYS(tag_range->start), PFN_PHYS(tag_range->end)); + ret = -EINVAL; + goto out_disabled; + } + } + + /* + * MTE disabled, tag storage pages can be used like any other pages. The + * only restriction is that the pages cannot be used by kexec because + * the memory remains marked as reserved in the memblock allocator. + */ + if (!system_supports_mte()) { + for (i = 0; i< num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + for (pfn = tag_range->start; pfn <= tag_range->end; pfn++) + free_reserved_page(pfn_to_page(pfn)); + } + ret = 0; + goto out_disabled; + } + + for (i = 0; i < num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + for (pfn = tag_range->start; pfn <= tag_range->end; pfn += pageblock_nr_pages) + init_cma_reserved_pageblock(pfn_to_page(pfn)); + totalcma_pages += range_len(tag_range); + } + + return 0; + +out_disabled: + pr_info("MTE tag storage region management disabled"); + return ret; +} +arch_initcall(mte_tag_storage_activate_regions); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 3c25226beeed..15f81429e145 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -83,6 +83,11 @@ static inline bool is_migrate_movable(int mt) return is_migrate_cma(mt) || mt == MIGRATE_MOVABLE; } +#ifdef CONFIG_CMA +/* Free whole pageblock and set its migration type to MIGRATE_CMA. */ +void init_cma_reserved_pageblock(struct page *page); +#endif + /* * Check whether a migratetype can be merged with another migratetype. * diff --git a/mm/internal.h b/mm/internal.h index b61034bd50f5..ddf6bb6c6308 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -539,9 +539,6 @@ isolate_migratepages_range(struct compact_control *cc, int __alloc_contig_migrate_range(struct compact_control *cc, unsigned long start, unsigned long end); -/* Free whole pageblock and set its migration type to MIGRATE_CMA. */ -void init_cma_reserved_pageblock(struct page *page); - #endif /* CONFIG_COMPACTION || CONFIG_CMA */ int find_suitable_fallback(struct free_area *area, unsigned int order, From patchwork Sun Nov 19 16:57:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460600 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B639DC5AE5B for ; Sun, 19 Nov 2023 16:59:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wTmCwtDxkyPPHG2fStQy/4KYMnMPZ77T2d3MNF+JOh8=; b=v8KEx2EcT6VF7Z lDFKd7akuKNVFG4+E9MfVXzKiKiIx0PuEO/JbhA7q3wP6xcV+dyALbln2OeC4zeW0fNKJUY0iBNAm SRFIFoco0E0Uo74bYi6Hrtid2rgvimuT5MB6iLfUta0d/09g39MozX/5YEVym+70pwJGHvKyKnlG/ Anq9tKQmPV5h0HVLMglzICe405bhxuR2m5jDvPozO+eHvX3dejTMAxnMErW+Ugv7WOSkksr9ToBVT DgPwOyroJ36UVM6AD2QZ/1sLpbbK2JxHbl+sfUGjMvk2lvdDvdbDfGwbMEjB6RGAJJEKdmjzx4BLr D2krbqfdTSUfoUfQ/VwQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8G-00AR4X-1I; Sun, 19 Nov 2023 16:58:52 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8C-00AR0Q-1j for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:50 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 528181477; Sun, 19 Nov 2023 08:59:34 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2807D3F6C4; Sun, 19 Nov 2023 08:58:43 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 13/27] arm64: mte: Make tag storage depend on ARCH_KEEP_MEMBLOCK Date: Sun, 19 Nov 2023 16:57:07 +0000 Message-Id: <20231119165721.9849-14-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085848_779321_8BDF0B1D X-CRM114-Status: GOOD ( 12.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Tag storage memory requires that the tag storage pages used for data are always migratable when they need to be repurposed to store tags. If ARCH_KEEP_MEMBLOCK is enabled, kexec will scan all non-reserved memblocks to find a suitable location for copying the kernel image. The kernel image, once loaded, cannot be moved to another location in physical memory. The initialization code for the tag storage reserves the memblocks for the tag storage pages, which means kexec will not use them, and the tag storage pages can be migrated at any time, which is the desired behaviour. However, if ARCH_KEEP_MEMBLOCK is not selected, kexec will not skip a region unless the memory resource has the IORESOURCE_SYSRAM_DRIVER_MANAGED flag, which isn't currently set by the tag storage initialization code. Make ARM64_MTE_TAG_STORAGE depend on ARCH_KEEP_MEMBLOCK to make it explicit that that the Kconfig option required for it to work correctly. Signed-off-by: Alexandru Elisei --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 047487046e8f..efa5b7958169 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2065,6 +2065,7 @@ config ARM64_MTE if ARM64_MTE config ARM64_MTE_TAG_STORAGE bool "Dynamic MTE tag storage management" + depends on ARCH_KEEP_MEMBLOCK select CONFIG_CMA help Adds support for dynamic management of the memory used by the hardware From patchwork Sun Nov 19 16:57:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460636 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9B6B6C072A2 for ; Sun, 19 Nov 2023 18:00:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WEXsvSa99pdAvzwr3bx63/3MxaBQbuOhHGv2Ev6icMk=; b=Kxou0bVoxW9C3i 4Vn657f8+4kFq5+lFwhcPmxHW2eLt6c5W9eXT4BEKqVMtUaXKYZSxWqMAjpP+3qfHuc89TyUerB5s Pcs/LOlD3F1dr9GFVX+ixxPX18eHi8+DmQintk3nwZvx378t2spyYg4DrI/88nV8vgiAs2K1rm8Y+ +4wvVXqSiUi6H7CKBSGYvbNvEE1L8ZAmtnW7AyYM1hotCSJv0/45sSb8NFlxQlUrBpq4sQ4vC6nJA 4DIBVO3h1x9umsYRRXJpg7jc24PuhCip2Dd0pyJysvBrF5XfPZPENQmVAMT12O667kptUpwH2SHiI 2BPz3YDCTqyH94TO54OQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4m5k-00AVvn-2F; Sun, 19 Nov 2023 18:00:20 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8H-00AR6a-2y for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:58:55 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8E171DA7; Sun, 19 Nov 2023 08:59:39 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5A3EA3F6C4; Sun, 19 Nov 2023 08:58:48 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 14/27] arm64: mte: Disable dynamic tag storage management if HW KASAN is enabled Date: Sun, 19 Nov 2023 16:57:08 +0000 Message-Id: <20231119165721.9849-15-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085854_182885_B560098A X-CRM114-Status: GOOD ( 13.54 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To be able to reserve the tag storage associated with a page requires that the tag storage page can be migrated. When HW KASAN is enabled, the kernel allocates pages, which are now tagged, in non-preemptible contexts, which can make reserving the associate tag storage impossible. Keep the tag storage pages reserved if HW KASAN is enabled. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 427f4f1909f3..8b9bedf7575d 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -308,6 +308,19 @@ static int __init mte_tag_storage_activate_regions(void) goto out_disabled; } + /* + * The kernel allocates memory in non-preemptible contexts, which makes + * migration impossible when reserving the associated tag storage. + * + * The check is safe to make because KASAN HW tags are enabled before + * the rest of the init functions are called, in smp_prepare_boot_cpu(). + */ + if (kasan_hw_tags_enabled()) { + pr_info("KASAN HW tags incompatible with MTE tag storage management"); + ret = 0; + goto out_disabled; + } + for (i = 0; i < num_tag_regions; i++) { tag_range = &tag_regions[i].tag_range; for (pfn = tag_range->start; pfn <= tag_range->end; pfn += pageblock_nr_pages) From patchwork Sun Nov 19 16:57:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C33A0C54E76 for ; Sun, 19 Nov 2023 16:59:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Q7fnU2hZwjOEkUkeOjc7WKM45BAXu5Qci5DUy6kh0ig=; b=enoGtW9YKYAdtY Z4mOmHyHGeJpXxvbJtPTUOjaCy5irXy4G4IilMuwQKi5lM4kEocyXnVySVIZynL7m5p9BJhcHZ6if 1BIbuBZR/WWsse8ag4ha3w2Y9pN+8LAbapLJm1DHk1anvaWcNC4JwddQxKuBlQ8XpLiIo8f5sRwaz tnK7e42OMFNBJly7cMTuGLvCdYcjN1fMc9MG5CLp9sYlVDm2K3gJK7HWImLJ+Qs78a523M++wm8j7 d+1c8bQIs+Ga63QOnCTkO8VcSlT7XvZhkB6rI1nsNGWAY+2ryUvNHRDMJn/bO5Z5tvI4gHSKXR/dC 7GsyiikzhSMSF2F86exg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8U-00ARHC-22; Sun, 19 Nov 2023 16:59:06 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8N-00ARAr-0p for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:02 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D2E19FEC; Sun, 19 Nov 2023 08:59:44 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9966B3F6C4; Sun, 19 Nov 2023 08:58:53 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 15/27] arm64: mte: Check that tag storage blocks are in the same zone Date: Sun, 19 Nov 2023 16:57:09 +0000 Message-Id: <20231119165721.9849-16-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085859_477698_B2552F8E X-CRM114-Status: GOOD ( 12.84 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org alloc_contig_range() requires that the requested pages are in the same zone. Check that this is indeed the case before initializing the tag storage blocks. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 33 +++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 8b9bedf7575d..fd63430d4dc0 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -265,6 +265,35 @@ void __init mte_tag_storage_init(void) } } +/* alloc_contig_range() requires all pages to be in the same zone. */ +static int __init mte_tag_storage_check_zone(void) +{ + struct range *tag_range; + struct zone *zone; + unsigned long pfn; + u32 block_size; + int i, j; + + for (i = 0; i < num_tag_regions; i++) { + block_size = tag_regions[i].block_size; + if (block_size == 1) + continue; + + tag_range = &tag_regions[i].tag_range; + for (pfn = tag_range->start; pfn <= tag_range->end; pfn += block_size) { + zone = page_zone(pfn_to_page(pfn)); + for (j = 1; j < block_size; j++) { + if (page_zone(pfn_to_page(pfn + j)) != zone) { + pr_err("Tag storage block pages in different zones"); + return -EINVAL; + } + } + } + } + + return 0; +} + static int __init mte_tag_storage_activate_regions(void) { phys_addr_t dram_start, dram_end; @@ -321,6 +350,10 @@ static int __init mte_tag_storage_activate_regions(void) goto out_disabled; } + ret = mte_tag_storage_check_zone(); + if (ret) + goto out_disabled; + for (i = 0; i < num_tag_regions; i++) { tag_range = &tag_regions[i].tag_range; for (pfn = tag_range->start; pfn <= tag_range->end; pfn += pageblock_nr_pages) From patchwork Sun Nov 19 16:57:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460602 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 06B7DC54E76 for ; Sun, 19 Nov 2023 16:59:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IkkvR/lElUV+dIxVA0VAWRtiXRYebtRKsatx7IOIiXc=; b=LkKbaSTzcPvKAV nfKJlpeZ8/hdr6CNhUsqiAlyLCSlpTVRZ2r2EpmVDEE4nlN+V5qTRUsdHOuK/TsS0CxwS1gnrkp/i YswWD0VGCSJw8bUEsXQ8M45wWRGojAmHrVsFCu1BVlDeW0Af7NJKpYQuBvpeGWgjmMR0nTP60G0MG tgyiHuqxfelqPtITZw1VzHVjLBVlFoBEPIHXdmFFKoA3mrP/8Egk0/W9W9N/GGSsMRT5h+ssk4iyd ljotxxOI0IHUZ2SWHkLzSpg9gnGxjqM6lGgCEifrdsDVTkVRGgtcYr9A2LEe0BHhLqFzxVDySWR1l S+Nh/j4nSyerQIAksoiA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8c-00AROf-0M; Sun, 19 Nov 2023 16:59:14 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8S-00ARFI-1c for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:07 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 245511480; Sun, 19 Nov 2023 08:59:50 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DBD283F6C4; Sun, 19 Nov 2023 08:58:58 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 16/27] arm64: mte: Manage tag storage on page allocation Date: Sun, 19 Nov 2023 16:57:10 +0000 Message-Id: <20231119165721.9849-17-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085904_747721_25CE0E4B X-CRM114-Status: GOOD ( 31.10 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Reserve tag storage for a tagged page by migrating the contents of the tag storage (if in use for data) and removing the tag storage pages from the page allocator by calling alloc_contig_range(). When all the associated tagged pages have been freed, return the tag storage pages back to the page allocator, where they can be used again for data allocations. Tag storage pages cannot be tagged, so disallow allocations from MIGRATE_CMA when the allocation is tagged. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte.h | 16 +- arch/arm64/include/asm/mte_tag_storage.h | 45 +++++ arch/arm64/include/asm/pgtable.h | 27 +++ arch/arm64/kernel/mte_tag_storage.c | 241 +++++++++++++++++++++++ fs/proc/page.c | 1 + include/linux/kernel-page-flags.h | 1 + include/linux/page-flags.h | 1 + include/trace/events/mmflags.h | 3 +- mm/huge_memory.c | 1 + 9 files changed, 333 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 8034695b3dd7..6457b7899207 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -40,12 +40,24 @@ void mte_free_tag_buf(void *buf); #ifdef CONFIG_ARM64_MTE /* track which pages have valid allocation tags */ -#define PG_mte_tagged PG_arch_2 +#define PG_mte_tagged PG_arch_2 /* simple lock to avoid multiple threads tagging the same page */ -#define PG_mte_lock PG_arch_3 +#define PG_mte_lock PG_arch_3 +/* Track if a tagged page has tag storage reserved */ +#define PG_tag_storage_reserved PG_arch_4 + +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +DECLARE_STATIC_KEY_FALSE(tag_storage_enabled_key); +extern bool page_tag_storage_reserved(struct page *page); +#endif static inline void set_page_mte_tagged(struct page *page) { +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE + /* Open code mte_tag_storage_enabled() */ + WARN_ON_ONCE(static_branch_likely(&tag_storage_enabled_key) && + !page_tag_storage_reserved(page)); +#endif /* * Ensure that the tags written prior to this function are visible * before the page flags update. diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index 8f86c4f9a7c3..cab033b184ab 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -5,11 +5,56 @@ #ifndef __ASM_MTE_TAG_STORAGE_H #define __ASM_MTE_TAG_STORAGE_H +#ifndef __ASSEMBLY__ + +#include + +#include + #ifdef CONFIG_ARM64_MTE_TAG_STORAGE + +DECLARE_STATIC_KEY_FALSE(tag_storage_enabled_key); + +static inline bool tag_storage_enabled(void) +{ + return static_branch_likely(&tag_storage_enabled_key); +} + +static inline bool alloc_requires_tag_storage(gfp_t gfp) +{ + return gfp & __GFP_TAGGED; +} + void mte_tag_storage_init(void); + +int reserve_tag_storage(struct page *page, int order, gfp_t gfp); +void free_tag_storage(struct page *page, int order); + +bool page_tag_storage_reserved(struct page *page); #else +static inline bool tag_storage_enabled(void) +{ + return false; +} +static inline bool alloc_requires_tag_storage(struct page *page) +{ + return false; +} static inline void mte_tag_storage_init(void) { } +static inline int reserve_tag_storage(struct page *page, int order, gfp_t gfp) +{ + return 0; +} +static inline void free_tag_storage(struct page *page, int order) +{ +} +static inline bool page_tag_storage_reserved(struct page *page) +{ + return true; +} #endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ + +#endif /* !__ASSEMBLY__ */ #endif /* __ASM_MTE_TAG_STORAGE_H */ diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index cd5dacd1be3a..20e8de853f5d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -10,6 +10,7 @@ #include #include +#include #include #include #include @@ -1063,6 +1064,32 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) mte_restore_page_tags_by_swp_entry(entry, &folio->page); } +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE + +#define __HAVE_ARCH_PREP_NEW_PAGE +static inline int arch_prep_new_page(struct page *page, int order, gfp_t gfp) +{ + if (tag_storage_enabled() && alloc_requires_tag_storage(gfp)) + return reserve_tag_storage(page, order, gfp); + return 0; +} + +#define __HAVE_ARCH_FREE_PAGES_PREPARE +static inline void arch_free_pages_prepare(struct page *page, int order) +{ + if (tag_storage_enabled() && page_mte_tagged(page)) + free_tag_storage(page, order); +} + +#define __HAVE_ARCH_ALLOC_CMA +static inline bool arch_alloc_cma(gfp_t gfp_mask) +{ + if (tag_storage_enabled() && alloc_requires_tag_storage(gfp_mask)) + return false; + return true; +} + +#endif /* CONFIG_ARM64_MTE_TAG_STORAGE */ #endif /* CONFIG_ARM64_MTE */ #define __HAVE_ARCH_CALC_VMA_GFP diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index fd63430d4dc0..9f8ef3116fc3 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -11,12 +11,18 @@ #include #include #include +#include +#include #include +#include #include +#include #include #include +__ro_after_init DEFINE_STATIC_KEY_FALSE(tag_storage_enabled_key); + struct tag_region { struct range mem_range; /* Memory associated with the tag storage, in PFNs. */ struct range tag_range; /* Tag storage memory, in PFNs. */ @@ -28,6 +34,31 @@ struct tag_region { static struct tag_region tag_regions[MAX_TAG_REGIONS]; static int num_tag_regions; +/* + * A note on locking. Reserving tag storage takes the tag_blocks_lock mutex, + * because alloc_contig_range() might sleep. + * + * Freeing tag storage takes the xa_lock spinlock with interrupts disabled + * because pages can be freed from non-preemptible contexts, including from an + * interrupt handler. + * + * Because tag storage can be freed from interrupt contexts, the xarray is + * defined with the XA_FLAGS_LOCK_IRQ flag to disable interrupts when calling + * xa_store(). This is done to prevent a deadlock with free_tag_storage() being + * called from an interrupt raised before xa_store() releases the xa_lock. + * + * All of the above means that reserve_tag_storage() cannot run concurrently + * with itself (no concurrent insertions), but it can run at the same time as + * free_tag_storage(). The first thing that reserve_tag_storage() does after + * taking the mutex is increase the refcount on all present tag storage blocks + * with the xa_lock held, to serialize against freeing the blocks. This is an + * optimization to avoid taking and releasing the xa_lock after each iteration + * if the refcount operation was moved inside the loop, where it would have had + * to be executed for each block. + */ +static DEFINE_XARRAY_FLAGS(tag_blocks_reserved, XA_FLAGS_LOCK_IRQ); +static DEFINE_MUTEX(tag_blocks_lock); + static int __init tag_storage_of_flat_get_range(unsigned long node, const __be32 *reg, int reg_len, struct range *range) { @@ -368,3 +399,213 @@ static int __init mte_tag_storage_activate_regions(void) return ret; } arch_initcall(mte_tag_storage_activate_regions); + +static void page_set_tag_storage_reserved(struct page *page, int order) +{ + int i; + + for (i = 0; i < (1 << order); i++) + set_bit(PG_tag_storage_reserved, &(page + i)->flags); +} + +static void block_ref_add(unsigned long block, struct tag_region *region, int order) +{ + int count; + + count = min(1u << order, 32 * region->block_size); + page_ref_add(pfn_to_page(block), count); +} + +static int block_ref_sub_return(unsigned long block, struct tag_region *region, int order) +{ + int count; + + count = min(1u << order, 32 * region->block_size); + return page_ref_sub_return(pfn_to_page(block), count); +} + +static bool tag_storage_block_is_reserved(unsigned long block) +{ + return xa_load(&tag_blocks_reserved, block) != NULL; +} + +static int tag_storage_reserve_block(unsigned long block, struct tag_region *region, int order) +{ + int ret; + + ret = xa_err(xa_store(&tag_blocks_reserved, block, pfn_to_page(block), GFP_KERNEL)); + if (!ret) + block_ref_add(block, region, order); + + return ret; +} + +static int order_to_num_blocks(int order) +{ + return max((1 << order) / 32, 1); +} + +static int tag_storage_find_block_in_region(struct page *page, unsigned long *blockp, + struct tag_region *region) +{ + struct range *tag_range = ®ion->tag_range; + struct range *mem_range = ®ion->mem_range; + u64 page_pfn = page_to_pfn(page); + u64 block, block_offset; + + if (!(mem_range->start <= page_pfn && page_pfn <= mem_range->end)) + return -ERANGE; + + block_offset = (page_pfn - mem_range->start) / 32; + block = tag_range->start + rounddown(block_offset, region->block_size); + + if (block + region->block_size - 1 > tag_range->end) { + pr_err("Block 0x%llx-0x%llx is outside tag region 0x%llx-0x%llx\n", + PFN_PHYS(block), PFN_PHYS(block + region->block_size), + PFN_PHYS(tag_range->start), PFN_PHYS(tag_range->end)); + return -ERANGE; + } + *blockp = block; + + return 0; + +} + +static int tag_storage_find_block(struct page *page, unsigned long *block, + struct tag_region **region) +{ + int i, ret; + + for (i = 0; i < num_tag_regions; i++) { + ret = tag_storage_find_block_in_region(page, block, &tag_regions[i]); + if (ret == 0) { + *region = &tag_regions[i]; + return 0; + } + } + + return -EINVAL; +} + +bool page_tag_storage_reserved(struct page *page) +{ + return test_bit(PG_tag_storage_reserved, &page->flags); +} + +int reserve_tag_storage(struct page *page, int order, gfp_t gfp) +{ + unsigned long start_block, end_block; + struct tag_region *region; + unsigned long block; + unsigned long flags; + unsigned int tries; + int ret = 0; + + VM_WARN_ON_ONCE(!preemptible()); + + if (page_tag_storage_reserved(page)) + return 0; + + /* + * __alloc_contig_migrate_range() ignores gfp when allocating the + * destination page for migration. Regardless, massage gfp flags and + * remove __GFP_TAGGED to avoid recursion in case gfp stops being + * ignored. + */ + gfp &= ~__GFP_TAGGED; + if (!(gfp & __GFP_NORETRY)) + gfp |= __GFP_RETRY_MAYFAIL; + + ret = tag_storage_find_block(page, &start_block, ®ion); + if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", page_to_pfn(page))) + return 0; + end_block = start_block + order_to_num_blocks(order) * region->block_size; + + mutex_lock(&tag_blocks_lock); + + /* Check again, this time with the lock held. */ + if (page_tag_storage_reserved(page)) + goto out_unlock; + + /* Make sure existing entries are not freed from out under out feet. */ + xa_lock_irqsave(&tag_blocks_reserved, flags); + for (block = start_block; block < end_block; block += region->block_size) { + if (tag_storage_block_is_reserved(block)) + block_ref_add(block, region, order); + } + xa_unlock_irqrestore(&tag_blocks_reserved, flags); + + for (block = start_block; block < end_block; block += region->block_size) { + /* Refcount incremented above. */ + if (tag_storage_block_is_reserved(block)) + continue; + + tries = 3; + while (tries--) { + ret = alloc_contig_range(block, block + region->block_size, MIGRATE_CMA, gfp); + if (ret == 0 || ret != -EBUSY) + break; + } + + if (ret) + goto out_error; + + ret = tag_storage_reserve_block(block, region, order); + if (ret) { + free_contig_range(block, region->block_size); + goto out_error; + } + + count_vm_events(CMA_ALLOC_SUCCESS, region->block_size); + } + + page_set_tag_storage_reserved(page, order); +out_unlock: + mutex_unlock(&tag_blocks_lock); + + return 0; + +out_error: + xa_lock_irqsave(&tag_blocks_reserved, flags); + for (block = start_block; block < end_block; block += region->block_size) { + if (tag_storage_block_is_reserved(block) && + block_ref_sub_return(block, region, order) == 1) { + __xa_erase(&tag_blocks_reserved, block); + free_contig_range(block, region->block_size); + } + } + xa_unlock_irqrestore(&tag_blocks_reserved, flags); + + mutex_unlock(&tag_blocks_lock); + + count_vm_events(CMA_ALLOC_FAIL, region->block_size); + + return ret; +} + +void free_tag_storage(struct page *page, int order) +{ + unsigned long block, start_block, end_block; + struct tag_region *region; + unsigned long flags; + int ret; + + ret = tag_storage_find_block(page, &start_block, ®ion); + if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", page_to_pfn(page))) + return; + + end_block = start_block + order_to_num_blocks(order) * region->block_size; + + xa_lock_irqsave(&tag_blocks_reserved, flags); + for (block = start_block; block < end_block; block += region->block_size) { + if (WARN_ONCE(!tag_storage_block_is_reserved(block), + "Block 0x%lx is not reserved for pfn 0x%lx", block, page_to_pfn(page))) + continue; + + if (block_ref_sub_return(block, region, order) == 1) { + __xa_erase(&tag_blocks_reserved, block); + free_contig_range(block, region->block_size); + } + } + xa_unlock_irqrestore(&tag_blocks_reserved, flags); +} diff --git a/fs/proc/page.c b/fs/proc/page.c index 195b077c0fac..e7eb584a9234 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -221,6 +221,7 @@ u64 stable_page_flags(struct page *page) #ifdef CONFIG_ARCH_USES_PG_ARCH_X u |= kpf_copy_bit(k, KPF_ARCH_2, PG_arch_2); u |= kpf_copy_bit(k, KPF_ARCH_3, PG_arch_3); + u |= kpf_copy_bit(k, KPF_ARCH_4, PG_arch_4); #endif return u; diff --git a/include/linux/kernel-page-flags.h b/include/linux/kernel-page-flags.h index 859f4b0c1b2b..4a0d719ffdd4 100644 --- a/include/linux/kernel-page-flags.h +++ b/include/linux/kernel-page-flags.h @@ -19,5 +19,6 @@ #define KPF_SOFTDIRTY 40 #define KPF_ARCH_2 41 #define KPF_ARCH_3 42 +#define KPF_ARCH_4 43 #endif /* LINUX_KERNEL_PAGE_FLAGS_H */ diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index a88e64acebfe..7915165a51bd 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -135,6 +135,7 @@ enum pageflags { #ifdef CONFIG_ARCH_USES_PG_ARCH_X PG_arch_2, PG_arch_3, + PG_arch_4, #endif __NR_PAGEFLAGS, diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 6ca0d5ed46c0..ba962fd10a2c 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -125,7 +125,8 @@ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ IF_HAVE_PG_IDLE(young) \ IF_HAVE_PG_ARCH_X(arch_2) \ -IF_HAVE_PG_ARCH_X(arch_3) +IF_HAVE_PG_ARCH_X(arch_3) \ +IF_HAVE_PG_ARCH_X(arch_4) #define show_page_flags(flags) \ (flags) ? __print_flags(flags, "|", \ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f31f02472396..9beead961a65 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2474,6 +2474,7 @@ static void __split_huge_page_tail(struct folio *folio, int tail, #ifdef CONFIG_ARCH_USES_PG_ARCH_X (1L << PG_arch_2) | (1L << PG_arch_3) | + (1L << PG_arch_4) | #endif (1L << PG_dirty) | LRU_GEN_MASK | LRU_REFS_MASK)); From patchwork Sun Nov 19 16:57:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E69C0C54E76 for ; Sun, 19 Nov 2023 16:59:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=a4xwCr4oH5Q1n+4U0BSilg7wisS1wR3slPxS2pXVZgM=; b=C+FEZUUmOJ6VcP 286q2WO1aV+A2MjowbZeANm6q4uhjjdKP7rfyc/oM7aReoOj8IY4202fANGPEVSkh/AILi/GhuAwx 2cJky4j6+iti37lGYKHc5tM+2XI5w3zOSU06ePb+fyLit0NjrVEcNbfKBmt6hawzBV10daOnfMhc+ oAkPQUFoBHclCffLrySoOmAV0HUu8AJjnkxx3hwD9wLwjbWtUbpo1tAVWrjsYWthpyxHi23Ru5ppi lxDJqjGmH14qcvd3HX9IRsk08cj99JF7T4EZIpBgejLZhZfZgVbe7rDa5fxOirXarcBIEZ0WlFipX xkJqX2CE4TMKgW54DA9Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8c-00ARPx-33; Sun, 19 Nov 2023 16:59:15 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8X-00ARJk-1m for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:11 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5E0701042; Sun, 19 Nov 2023 08:59:55 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 2A15B3F6C4; Sun, 19 Nov 2023 08:59:04 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 17/27] arm64: mte: Perform CMOs for tag blocks on tagged page allocation/free Date: Sun, 19 Nov 2023 16:57:11 +0000 Message-Id: <20231119165721.9849-18-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085909_749392_E02677E7 X-CRM114-Status: GOOD ( 13.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Make sure the contents of the tag storage block is not corrupted by performing: 1. A tag dcache inval when the associated tagged pages are freed, to avoid dirty tag cache lines being evicted and corrupting the tag storage block when it's being used to store data. 2. A data cache inval when the tag storage block is being reserved, to ensure that no dirty data cache lines are present, which would trigger a writeback that could corrupt the tags stored in the block. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/assembler.h | 10 ++++++++++ arch/arm64/include/asm/mte_tag_storage.h | 2 ++ arch/arm64/kernel/mte_tag_storage.c | 11 +++++++++++ arch/arm64/lib/mte.S | 16 ++++++++++++++++ 4 files changed, 39 insertions(+) diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h index 376a980f2bad..8d41c8cfdc69 100644 --- a/arch/arm64/include/asm/assembler.h +++ b/arch/arm64/include/asm/assembler.h @@ -310,6 +310,16 @@ alternative_cb_end lsl \reg, \reg, \tmp // actual cache line size .endm +/* + * tcache_line_size - get the safe tag cache line size across all CPUs + */ + .macro tcache_line_size, reg, tmp + read_ctr \tmp + ubfm \tmp, \tmp, #32, #37 // tag cache line size encoding + mov \reg, #4 // bytes per word + lsl \reg, \reg, \tmp // actual tag cache line size + .endm + /* * raw_icache_line_size - get the minimum I-cache line size on this CPU * from the CTR register. diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index cab033b184ab..6e5d28e607bb 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -11,6 +11,8 @@ #include +extern void dcache_inval_tags_poc(unsigned long start, unsigned long end); + #ifdef CONFIG_ARM64_MTE_TAG_STORAGE DECLARE_STATIC_KEY_FALSE(tag_storage_enabled_key); diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 9f8ef3116fc3..833480048170 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -19,6 +19,7 @@ #include #include +#include #include __ro_after_init DEFINE_STATIC_KEY_FALSE(tag_storage_enabled_key); @@ -431,8 +432,13 @@ static bool tag_storage_block_is_reserved(unsigned long block) static int tag_storage_reserve_block(unsigned long block, struct tag_region *region, int order) { + unsigned long block_va; int ret; + block_va = (unsigned long)page_to_virt(pfn_to_page(block)); + /* Avoid writeback of dirty data cache lines corrupting tags. */ + dcache_inval_poc(block_va, block_va + region->block_size * PAGE_SIZE); + ret = xa_err(xa_store(&tag_blocks_reserved, block, pfn_to_page(block), GFP_KERNEL)); if (!ret) block_ref_add(block, region, order); @@ -587,6 +593,7 @@ void free_tag_storage(struct page *page, int order) { unsigned long block, start_block, end_block; struct tag_region *region; + unsigned long page_va; unsigned long flags; int ret; @@ -594,6 +601,10 @@ void free_tag_storage(struct page *page, int order) if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", page_to_pfn(page))) return; + page_va = (unsigned long)page_to_virt(page); + /* Avoid writeback of dirty tag cache lines corrupting data. */ + dcache_inval_tags_poc(page_va, page_va + (PAGE_SIZE << order)); + end_block = start_block + order_to_num_blocks(order) * region->block_size; xa_lock_irqsave(&tag_blocks_reserved, flags); diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 9f623e9da09f..bc02b4e95062 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -175,3 +175,19 @@ SYM_FUNC_START(mte_copy_page_tags_from_buf) ret SYM_FUNC_END(mte_copy_page_tags_from_buf) + +/* + * dcache_inval_tags_poc(start, end) + * + * Ensure that any tags in the D-cache for the interval [start, end) + * are invalidated to PoC. + * + * - start - virtual start address of region + * - end - virtual end address of region + */ +SYM_FUNC_START(__pi_dcache_inval_tags_poc) + tcache_line_size x2, x3 + dcache_by_myline_op igvac, sy, x0, x1, x2, x3 + ret +SYM_FUNC_END(__pi_dcache_inval_tags_poc) +SYM_FUNC_ALIAS(dcache_inval_tags_poc, __pi_dcache_inval_tags_poc) From patchwork Sun Nov 19 16:57:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460604 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2096C072A2 for ; Sun, 19 Nov 2023 16:59:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7YHpZvZG47Ol11mnMyxFPgxfiN1p4dLZBIM0MNyj2xY=; b=W9OCP5Jcwp9zA3 yS69ji8CD0udfkyYmrHBiruIVV9aKzdbTMEVoClnBlgSECI6igJ65ubeBNHaGKSNfrfaPvhoYMZby qsZRueLDIypfMml3Ak3pjEnGOQJYBADTjYMvpLO5gFo1bsuxeVFVIUry+XjNgHPhr3VNXT3stXZ7u 2BDOp3wP0B/yDDWp5osaLjZM8wzWBwSeSuUyt6MpfYbtkE9DWLxGk6UDhOFamCNYUh70uzXAOpyp1 apVpG+2bvrV+wcTlipcgKVTdZMcFU3n48q6zw6BGqx1vXCMcmgSLAj9kSyrRM24NcmB8mdcZSXgeN VQzUELPaRUB1kpyJbabw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8m-00ARYc-2P; Sun, 19 Nov 2023 16:59:25 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8c-00ARPq-2t for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:16 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 90D471007; Sun, 19 Nov 2023 09:00:00 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 64C553F6C4; Sun, 19 Nov 2023 08:59:09 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 18/27] arm64: mte: Reserve tag block for the zero page Date: Sun, 19 Nov 2023 16:57:12 +0000 Message-Id: <20231119165721.9849-19-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085914_977334_C618DE32 X-CRM114-Status: UNSURE ( 9.69 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On arm64, the zero page receives special treatment by having the tagged flag set on MTE initialization, not when the page is mapped in a process address space. Reserve the corresponding tag block when tag storage management is being activated. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 833480048170..a1cc239f7211 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -393,6 +393,8 @@ static int __init mte_tag_storage_activate_regions(void) totalcma_pages += range_len(tag_range); } + reserve_tag_storage(ZERO_PAGE(0), 0, GFP_HIGHUSER_MOVABLE); + return 0; out_disabled: From patchwork Sun Nov 19 16:57:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 93CFEC54E76 for ; Sun, 19 Nov 2023 17:00:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=FKuaGXTmj/4mO/CF4sCAMoB5iLvVOKbfhkZ+bO610ww=; b=UnxqWO7VNG2xHO hEUL+VExystlhJTgEtG0PUxigncRf0AxkjljA6d+nqlLcQcvkQQd7KbOHlM4vchFuLz/0lcTCfTQw z2Nvce5D9hBNPKuNbJ6A/WEwSBx05gZWo4/f/e34L5mRUD3nPgjACUGrMKINHS2sPvc80YuRab0af lXMe6TtxDek35/hzMt3eTIae+9McMVDlvqr8aYu5688CB6v1z10RpOpAyGGYSYwoF4/POFiaAs2eI M1oqppaee4GBExC+4ZuTk6ERewQVy858scm8RQyI67CnZFZjEcMOUyYQEC1a+PMw2k7/dKDuTcN7c VjfNLphf0ruZXsJTZfdQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8t-00ARep-2n; Sun, 19 Nov 2023 16:59:31 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8i-00ARV0-0d for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:24 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C6B291476; Sun, 19 Nov 2023 09:00:05 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 98F883F6C4; Sun, 19 Nov 2023 08:59:14 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 19/27] mm: mprotect: Introduce PAGE_FAULT_ON_ACCESS for mprotect(PROT_MTE) Date: Sun, 19 Nov 2023 16:57:13 +0000 Message-Id: <20231119165721.9849-20-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085920_832154_E2AFEFC5 X-CRM114-Status: GOOD ( 31.47 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org To enable tagging on a memory range, userspace can use mprotect() with the PROT_MTE access flag. Pages already mapped in the VMA don't have the associated tag storage block reserved, so mark the PTEs as PAGE_FAULT_ON_ACCESS to trigger a fault next time they are accessed, and reserve the tag storage on the fault path. This has several benefits over reserving the tag storage as part of the mprotect() call handling: - Tag storage is reserved only for those pages in the VMA that are accessed, instead of for all the pages already mapped in the VMA. - Reduces the latency of the mprotect() call. - Eliminates races with page migration. But all of this is at the expense of an extra page fault per page until the pages being accessed all have their corresponding tag storage reserved. For arm64, the PAGE_FAULT_ON_ACCESS protection is created by defining a new page table entry software bit, PTE_TAG_STORAGE_NONE. Linux doesn't set any of the PBHA bits in entries from the last level of the translation table and it doesn't use the TCR_ELx.HWUxx bits; also, the first PBHA bit, bit 59, is already being used as a software bit for PMD_PRESENT_INVALID. This is only implemented for PTE mappings; PMD mappings will follow. Signed-off-by: Alexandru Elisei --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/mte.h | 4 +- arch/arm64/include/asm/mte_tag_storage.h | 2 + arch/arm64/include/asm/pgtable-prot.h | 2 + arch/arm64/include/asm/pgtable.h | 40 ++++++--- arch/arm64/kernel/mte.c | 12 ++- arch/arm64/mm/fault.c | 101 +++++++++++++++++++++++ include/linux/pgtable.h | 17 ++++ mm/Kconfig | 3 + mm/memory.c | 3 + 10 files changed, 170 insertions(+), 15 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index efa5b7958169..3b9c435eaafb 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2066,6 +2066,7 @@ if ARM64_MTE config ARM64_MTE_TAG_STORAGE bool "Dynamic MTE tag storage management" depends on ARCH_KEEP_MEMBLOCK + select ARCH_HAS_FAULT_ON_ACCESS select CONFIG_CMA help Adds support for dynamic management of the memory used by the hardware diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 6457b7899207..70dc2e409070 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -107,7 +107,7 @@ static inline bool try_page_mte_tagging(struct page *page) } void mte_zero_clear_page_tags(void *addr); -void mte_sync_tags(pte_t pte, unsigned int nr_pages); +void mte_sync_tags(pte_t *pteval, unsigned int nr_pages); void mte_copy_page_tags(void *kto, const void *kfrom); void mte_thread_init_user(void); void mte_thread_switch(struct task_struct *next); @@ -139,7 +139,7 @@ static inline bool try_page_mte_tagging(struct page *page) static inline void mte_zero_clear_page_tags(void *addr) { } -static inline void mte_sync_tags(pte_t pte, unsigned int nr_pages) +static inline void mte_sync_tags(pte_t *pteval, unsigned int nr_pages) { } static inline void mte_copy_page_tags(void *kto, const void *kfrom) diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index 6e5d28e607bb..c70ced60a0cd 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -33,6 +33,8 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp); void free_tag_storage(struct page *page, int order); bool page_tag_storage_reserved(struct page *page); + +vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf); #else static inline bool tag_storage_enabled(void) { diff --git a/arch/arm64/include/asm/pgtable-prot.h b/arch/arm64/include/asm/pgtable-prot.h index e9624f6326dd..85ebb3e352ad 100644 --- a/arch/arm64/include/asm/pgtable-prot.h +++ b/arch/arm64/include/asm/pgtable-prot.h @@ -19,6 +19,7 @@ #define PTE_SPECIAL (_AT(pteval_t, 1) << 56) #define PTE_DEVMAP (_AT(pteval_t, 1) << 57) #define PTE_PROT_NONE (_AT(pteval_t, 1) << 58) /* only when !PTE_VALID */ +#define PTE_TAG_STORAGE_NONE (_AT(pteval_t, 1) << 60) /* only when PTE_PROT_NONE */ /* * This bit indicates that the entry is present i.e. pmd_page() @@ -94,6 +95,7 @@ extern bool arm64_use_ng_mappings; }) #define PAGE_NONE __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) +#define PAGE_FAULT_ON_ACCESS __pgprot(((_PAGE_DEFAULT) & ~PTE_VALID) | PTE_PROT_NONE | PTE_TAG_STORAGE_NONE | PTE_RDONLY | PTE_NG | PTE_PXN | PTE_UXN) /* shared+writable pages are clean by default, hence PTE_RDONLY|PTE_WRITE */ #define PAGE_SHARED __pgprot(_PAGE_SHARED) #define PAGE_SHARED_EXEC __pgprot(_PAGE_SHARED_EXEC) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 20e8de853f5d..8cc135f1c112 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -326,10 +326,10 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, __func__, pte_val(old_pte), pte_val(pte)); } -static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) +static inline void __sync_cache_and_tags(pte_t *pteval, unsigned int nr_pages) { - if (pte_present(pte) && pte_user_exec(pte) && !pte_special(pte)) - __sync_icache_dcache(pte); + if (pte_present(*pteval) && pte_user_exec(*pteval) && !pte_special(*pteval)) + __sync_icache_dcache(*pteval); /* * If the PTE would provide user space access to the tags associated @@ -337,9 +337,9 @@ static inline void __sync_cache_and_tags(pte_t pte, unsigned int nr_pages) * pte_access_permitted() returns false for exec only mappings, they * don't expose tags (instruction fetches don't check tags). */ - if (system_supports_mte() && pte_access_permitted(pte, false) && - !pte_special(pte) && pte_tagged(pte)) - mte_sync_tags(pte, nr_pages); + if (system_supports_mte() && pte_access_permitted(*pteval, false) && + !pte_special(*pteval) && pte_tagged(*pteval)) + mte_sync_tags(pteval, nr_pages); } static inline void set_ptes(struct mm_struct *mm, @@ -347,7 +347,7 @@ static inline void set_ptes(struct mm_struct *mm, pte_t *ptep, pte_t pte, unsigned int nr) { page_table_check_ptes_set(mm, ptep, pte, nr); - __sync_cache_and_tags(pte, nr); + __sync_cache_and_tags(&pte, nr); for (;;) { __check_safe_pte_update(mm, ptep, pte); @@ -459,6 +459,26 @@ static inline int pmd_protnone(pmd_t pmd) } #endif +#ifdef CONFIG_ARCH_HAS_FAULT_ON_ACCESS +static inline bool fault_on_access_pte(pte_t pte) +{ + return (pte_val(pte) & (PTE_PROT_NONE | PTE_TAG_STORAGE_NONE | PTE_VALID)) == + (PTE_PROT_NONE | PTE_TAG_STORAGE_NONE); +} + +static inline bool fault_on_access_pmd(pmd_t pmd) +{ + return fault_on_access_pte(pmd_pte(pmd)); +} + +static inline vm_fault_t arch_do_page_fault_on_access(struct vm_fault *vmf) +{ + if (tag_storage_enabled()) + return handle_page_missing_tag_storage(vmf); + return VM_FAULT_SIGBUS; +} +#endif /* CONFIG_ARCH_HAS_FAULT_ON_ACCESS */ + #define pmd_present_invalid(pmd) (!!(pmd_val(pmd) & PMD_PRESENT_INVALID)) static inline int pmd_present(pmd_t pmd) @@ -533,7 +553,7 @@ static inline void __set_pte_at(struct mm_struct *mm, unsigned long __always_unused addr, pte_t *ptep, pte_t pte, unsigned int nr) { - __sync_cache_and_tags(pte, nr); + __sync_cache_and_tags(&pte, nr); __check_safe_pte_update(mm, ptep, pte); set_pte(ptep, pte); } @@ -828,8 +848,8 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot) * in MAIR_EL1. The mask below has to include PTE_ATTRINDX_MASK. */ const pteval_t mask = PTE_USER | PTE_PXN | PTE_UXN | PTE_RDONLY | - PTE_PROT_NONE | PTE_VALID | PTE_WRITE | PTE_GP | - PTE_ATTRINDX_MASK; + PTE_PROT_NONE | PTE_TAG_STORAGE_NONE | PTE_VALID | + PTE_WRITE | PTE_GP | PTE_ATTRINDX_MASK; /* preserve the hardware dirty information */ if (pte_hw_dirty(pte)) pte = set_pte_bit(pte, __pgprot(PTE_DIRTY)); diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index a41ef3213e1e..5962bab1d549 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include @@ -35,13 +36,18 @@ DEFINE_STATIC_KEY_FALSE(mte_async_or_asymm_mode); EXPORT_SYMBOL_GPL(mte_async_or_asymm_mode); #endif -void mte_sync_tags(pte_t pte, unsigned int nr_pages) +void mte_sync_tags(pte_t *pteval, unsigned int nr_pages) { - struct page *page = pte_page(pte); + struct page *page = pte_page(*pteval); unsigned int i; - /* if PG_mte_tagged is set, tags have already been initialised */ for (i = 0; i < nr_pages; i++, page++) { + if (tag_storage_enabled() && unlikely(!page_tag_storage_reserved(page))) { + *pteval = pte_modify(*pteval, PAGE_FAULT_ON_ACCESS); + continue; + } + + /* if PG_mte_tagged is set, tags have already been initialised */ if (try_page_mte_tagging(page)) { mte_clear_page_tags(page_address(page)); set_page_mte_tagged(page); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index acbc7530d2b2..f5fa583acf18 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include #include @@ -953,3 +954,103 @@ void tag_clear_highpage(struct page *page) mte_zero_clear_page_tags(page_address(page)); set_page_mte_tagged(page); } + +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf) +{ + struct vm_area_struct *vma = vmf->vma; + struct page *page = NULL; + pte_t new_pte, old_pte; + bool writable = false; + vm_fault_t err; + int ret; + + spin_lock(vmf->ptl); + if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; + } + + /* Get the normal PTE */ + old_pte = ptep_get(vmf->pte); + new_pte = pte_modify(old_pte, vma->vm_page_prot); + + /* + * Detect now whether the PTE could be writable; this information + * is only valid while holding the PT lock. + */ + writable = pte_write(new_pte); + if (!writable && vma_wants_manual_pte_write_upgrade(vma) && + can_change_pte_writable(vma, vmf->address, new_pte)) + writable = true; + + page = vm_normal_page(vma, vmf->address, new_pte); + if (!page || is_zone_device_page(page)) + goto out_map; + + /* + * This should never happen, once a VMA has been marked as tagged, that + * cannot be changed. + */ + if (!(vma->vm_flags & VM_MTE)) + goto out_map; + + /* Prevent the page from being unmapped from under us. */ + get_page(page); + vma_set_access_pid_bit(vma); + + /* + * Pairs with pte_offset_map_nolock(), which takes the RCU read lock, + * and spin_lock() above which takes the ptl lock. Both locks should be + * balanced after this point. + */ + pte_unmap_unlock(vmf->pte, vmf->ptl); + + /* + * Probably the page is being isolated for migration, replay the fault + * to give time for the entry to be replaced by a migration pte. + */ + if (unlikely(is_migrate_isolate_page(page))) + goto out_retry; + + ret = reserve_tag_storage(page, 0, GFP_HIGHUSER_MOVABLE); + if (ret) + goto out_retry; + + put_page(page); + + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + return 0; + } + +out_map: + /* + * Make it present again, depending on how arch implements + * non-accessible ptes, some can allow access by kernel mode. + */ + old_pte = ptep_modify_prot_start(vma, vmf->address, vmf->pte); + new_pte = pte_modify(old_pte, vma->vm_page_prot); + new_pte = pte_mkyoung(new_pte); + if (writable) + new_pte = pte_mkwrite(new_pte, vma); + ptep_modify_prot_commit(vma, vmf->address, vmf->pte, old_pte, new_pte); + update_mmu_cache(vma, vmf->address, vmf->pte); + pte_unmap_unlock(vmf->pte, vmf->ptl); + + return 0; + +out_retry: + put_page(page); + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + vma_end_read(vma); + if (fault_flag_allow_retry_first(vmf->flags)) { + err = VM_FAULT_RETRY; + } else { + /* Replay the fault. */ + err = 0; + } + return err; +} +#endif diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index ffdb9b6bed6c..e2c761dd6c41 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1458,6 +1458,23 @@ static inline int pmd_protnone(pmd_t pmd) } #endif /* CONFIG_NUMA_BALANCING */ +#ifndef CONFIG_ARCH_HAS_FAULT_ON_ACCESS +static inline bool fault_on_access_pte(pte_t pte) +{ + return false; +} + +static inline bool fault_on_access_pmd(pmd_t pmd) +{ + return false; +} + +static inline vm_fault_t arch_do_page_fault_on_access(struct vm_fault *vmf) +{ + return VM_FAULT_SIGBUS; +} +#endif + #endif /* CONFIG_MMU */ #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP diff --git a/mm/Kconfig b/mm/Kconfig index 89971a894b60..a90eefc3ee80 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1019,6 +1019,9 @@ config IDLE_PAGE_TRACKING config ARCH_HAS_CACHE_LINE_SIZE bool +config ARCH_HAS_FAULT_ON_ACCESS + bool + config ARCH_HAS_CURRENT_STACK_POINTER bool help diff --git a/mm/memory.c b/mm/memory.c index e137f7673749..a04a971200b9 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5044,6 +5044,9 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (!pte_present(vmf->orig_pte)) return do_swap_page(vmf); + if (fault_on_access_pte(vmf->orig_pte) && vma_is_accessible(vmf->vma)) + return arch_do_page_fault_on_access(vmf); + if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); From patchwork Sun Nov 19 16:57:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460606 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91336C072A2 for ; Sun, 19 Nov 2023 17:00:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=gViwXi00I4Q6OrwyasuRVyLzmGtpBCxgLleZDyCbEvo=; b=1T54jEnXBasTiH 0HKJ45Ns+g1mNlHECQC2I909BsGtw0udD3VEYHG+NK+iV7+m39MpWh+IuVneC3FHvqNorost/CKV7 GIFwy88xKJiddKkEQ7g5rjRuz3R3MZ3nCL1eyQ8bGrbWRx1m9Sw3I0sSbbOwDvvIyu28flnn1DB09 PDqP3hiL0HTB013FeAHNK+01GUnZ/KI8ZTrokcPEh7bUCvfsRmC3DGZZ7eFYe82chjX629q7Hsdlg YsBMSo8x+xw/e7o8SQckkp5x51VScZUd5jlblahf6IFKIVmAfUWVyjCYURvF5AkDa+yy2kp961I8A 1S8QjlfJeQebedrG014A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8x-00ARhr-0I; Sun, 19 Nov 2023 16:59:35 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8n-00ARZ7-1F for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:27 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 182CE1477; Sun, 19 Nov 2023 09:00:11 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D31763F6C4; Sun, 19 Nov 2023 08:59:19 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 20/27] mm: hugepage: Handle huge page fault on access Date: Sun, 19 Nov 2023 16:57:14 +0000 Message-Id: <20231119165721.9849-21-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085925_621032_A6BF6B70 X-CRM114-Status: GOOD ( 16.61 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Handle PAGE_FAULT_ON_ACCESS faults for huge pages in a similar way to regular pages. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte_tag_storage.h | 1 + arch/arm64/include/asm/pgtable.h | 7 ++ arch/arm64/mm/fault.c | 81 ++++++++++++++++++++++++ include/linux/huge_mm.h | 2 + include/linux/pgtable.h | 5 ++ mm/huge_memory.c | 4 +- mm/memory.c | 3 + 7 files changed, 101 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index c70ced60a0cd..b97406d369ce 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -35,6 +35,7 @@ void free_tag_storage(struct page *page, int order); bool page_tag_storage_reserved(struct page *page); vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf); +vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf); #else static inline bool tag_storage_enabled(void) { diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 8cc135f1c112..1704411c096d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -477,6 +477,13 @@ static inline vm_fault_t arch_do_page_fault_on_access(struct vm_fault *vmf) return handle_page_missing_tag_storage(vmf); return VM_FAULT_SIGBUS; } + +static inline vm_fault_t arch_do_huge_page_fault_on_access(struct vm_fault *vmf) +{ + if (tag_storage_enabled()) + return handle_huge_page_missing_tag_storage(vmf); + return VM_FAULT_SIGBUS; +} #endif /* CONFIG_ARCH_HAS_FAULT_ON_ACCESS */ #define pmd_present_invalid(pmd) (!!(pmd_val(pmd) & PMD_PRESENT_INVALID)) diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index f5fa583acf18..6730a0812a24 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -1041,6 +1041,87 @@ vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf) return 0; +out_retry: + put_page(page); + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + vma_end_read(vma); + if (fault_flag_allow_retry_first(vmf->flags)) { + err = VM_FAULT_RETRY; + } else { + /* Replay the fault. */ + err = 0; + } + return err; +} + +vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf) +{ + unsigned long haddr = vmf->address & HPAGE_PMD_MASK; + struct vm_area_struct *vma = vmf->vma; + pmd_t old_pmd, new_pmd; + bool writable = false; + struct page *page; + vm_fault_t err; + int ret; + + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); + if (unlikely(!pmd_same(vmf->orig_pmd, *vmf->pmd))) { + spin_unlock(vmf->ptl); + return 0; + } + + old_pmd = vmf->orig_pmd; + new_pmd = pmd_modify(old_pmd, vma->vm_page_prot); + + /* + * Detect now whether the PMD could be writable; this information + * is only valid while holding the PT lock. + */ + writable = pmd_write(new_pmd); + if (!writable && vma_wants_manual_pte_write_upgrade(vma) && + can_change_pmd_writable(vma, vmf->address, new_pmd)) + writable = true; + + page = vm_normal_page_pmd(vma, haddr, new_pmd); + if (!page) + goto out_map; + + if (!(vma->vm_flags & VM_MTE)) + goto out_map; + + get_page(page); + vma_set_access_pid_bit(vma); + + spin_unlock(vmf->ptl); + writable = false; + + if (unlikely(is_migrate_isolate_page(page))) + goto out_retry; + + ret = reserve_tag_storage(page, HPAGE_PMD_ORDER, GFP_HIGHUSER_MOVABLE); + if (ret) + goto out_retry; + + put_page(page); + + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); + if (unlikely(!pmd_same(old_pmd, *vmf->pmd))) { + spin_unlock(vmf->ptl); + return 0; + } + +out_map: + /* Restore the PMD */ + new_pmd = pmd_modify(old_pmd, vma->vm_page_prot); + new_pmd = pmd_mkyoung(new_pmd); + if (writable) + new_pmd = pmd_mkwrite(new_pmd, vma); + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, new_pmd); + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd); + spin_unlock(vmf->ptl); + + return 0; + out_retry: put_page(page); if (vmf->flags & FAULT_FLAG_VMA_LOCK) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fa0350b0812a..bb84291f9231 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -36,6 +36,8 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, pgprot_t newprot, unsigned long cp_flags); +bool can_change_pmd_writable(struct vm_area_struct *vma, unsigned long addr, + pmd_t pmd); vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, pfn_t pfn, bool write); vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, pfn_t pfn, bool write); diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e2c761dd6c41..de45f475bf8d 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1473,6 +1473,11 @@ static inline vm_fault_t arch_do_page_fault_on_access(struct vm_fault *vmf) { return VM_FAULT_SIGBUS; } + +static inline vm_fault_t arch_do_huge_page_fault_on_access(struct vm_fault *vmf) +{ + return VM_FAULT_SIGBUS; +} #endif #endif /* CONFIG_MMU */ diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9beead961a65..d1402b43ea39 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1406,8 +1406,8 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) return VM_FAULT_FALLBACK; } -static inline bool can_change_pmd_writable(struct vm_area_struct *vma, - unsigned long addr, pmd_t pmd) +inline bool can_change_pmd_writable(struct vm_area_struct *vma, + unsigned long addr, pmd_t pmd) { struct page *page; diff --git a/mm/memory.c b/mm/memory.c index a04a971200b9..46b926625503 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5168,6 +5168,9 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, return 0; } if (pmd_trans_huge(vmf.orig_pmd) || pmd_devmap(vmf.orig_pmd)) { + if (fault_on_access_pmd(vmf.orig_pmd) && vma_is_accessible(vma)) + return arch_do_huge_page_fault_on_access(&vmf); + if (pmd_protnone(vmf.orig_pmd) && vma_is_accessible(vma)) return do_huge_pmd_numa_page(&vmf); From patchwork Sun Nov 19 16:57:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 72A2AC072A2 for ; Sun, 19 Nov 2023 17:00:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=X1P7kdqboSyTPqZMAiZoubPVsmUUB6Zp6O6pBNGPllQ=; b=o97N8xLTXncq9I JX79OtRK9qjeJ3Pm0xUquFMkH20Dx1lvfkRH3WEIwVtbVxSYpo2MpB8hxdDm110YUqz/yJ1AYQ8+R gEOn+Mn/hyiB8gS6My0Zo7BREKzEX6wQQCKzhdrUeA3lX6zzPn4EWYXbKJxGO1MAHV2Csvq42QsfC aaYCX/bQsWZfq1zUSgTgmJK7qw4uDZ+Xq1tH5ppMDPBcLKpHQAXaTUBohY5LASPMsx14g53Ccivpi opkvn1n7wGqs2Eq5YPN8Ty8gQmeDRt7zvZfVbELdR1uNVaz3DL/J8l9/uvGt9r+NgS6yA7mIZcQB2 iqf1JqzOs3VBGRoBOkfg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l94-00ARql-3C; Sun, 19 Nov 2023 16:59:43 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8s-00ARdX-2e for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:32 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6BE0DDA7; Sun, 19 Nov 2023 09:00:16 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 21D8F3F6C4; Sun, 19 Nov 2023 08:59:25 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 21/27] mm: arm64: Handle tag storage pages mapped before mprotect(PROT_MTE) Date: Sun, 19 Nov 2023 16:57:15 +0000 Message-Id: <20231119165721.9849-22-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085931_024666_8BFA0CC0 X-CRM114-Status: GOOD ( 17.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte_tag_storage.h | 1 + arch/arm64/kernel/mte_tag_storage.c | 15 +++++++ arch/arm64/mm/fault.c | 55 ++++++++++++++++++++++++ include/linux/migrate.h | 8 +++- include/linux/migrate_mode.h | 1 + mm/internal.h | 6 --- 6 files changed, 78 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index b97406d369ce..6a8b19a6a758 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -33,6 +33,7 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp); void free_tag_storage(struct page *page, int order); bool page_tag_storage_reserved(struct page *page); +bool page_is_tag_storage(struct page *page); vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf); vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf); diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index a1cc239f7211..5096ce859136 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -500,6 +500,21 @@ bool page_tag_storage_reserved(struct page *page) return test_bit(PG_tag_storage_reserved, &page->flags); } +bool page_is_tag_storage(struct page *page) +{ + unsigned long pfn = page_to_pfn(page); + struct range *tag_range; + int i; + + for (i = 0; i < num_tag_regions; i++) { + tag_range = &tag_regions[i].tag_range; + if (tag_range->start <= pfn && pfn <= tag_range->end) + return true; + } + + return false; +} + int reserve_tag_storage(struct page *page, int order, gfp_t gfp) { unsigned long start_block, end_block; diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 6730a0812a24..964c5ae161a3 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -956,6 +957,50 @@ void tag_clear_highpage(struct page *page) } #ifdef CONFIG_ARM64_MTE_TAG_STORAGE + +#define MR_TAGGED_TAG_STORAGE MR_ARCH_1 + +extern bool isolate_lru_page(struct page *page); +extern void putback_movable_pages(struct list_head *l); + +/* Returns with the page reference dropped. */ +static void migrate_tag_storage_page(struct page *page) +{ + struct migration_target_control mtc = { + .nid = NUMA_NO_NODE, + .gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_TAGGED, + }; + unsigned long i, nr_pages = compound_nr(page); + LIST_HEAD(pagelist); + int ret, tries; + + lru_cache_disable(); + + for (i = 0; i < nr_pages; i++) { + if (!isolate_lru_page(page + i)) { + ret = -EAGAIN; + goto out; + } + /* Isolate just grabbed another reference, drop ours. */ + put_page(page + i); + list_add_tail(&(page + i)->lru, &pagelist); + } + + tries = 5; + while (tries--) { + ret = migrate_pages(&pagelist, alloc_migration_target, NULL, (unsigned long)&mtc, + MIGRATE_SYNC, MR_TAGGED_TAG_STORAGE, NULL); + if (ret == 0 || ret != -EBUSY) + break; + } + +out: + if (ret != 0) + putback_movable_pages(&pagelist); + + lru_cache_enable(); +} + vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; @@ -1013,6 +1058,11 @@ vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf) if (unlikely(is_migrate_isolate_page(page))) goto out_retry; + if (unlikely(page_is_tag_storage(page))) { + migrate_tag_storage_page(page); + return 0; + } + ret = reserve_tag_storage(page, 0, GFP_HIGHUSER_MOVABLE); if (ret) goto out_retry; @@ -1098,6 +1148,11 @@ vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf) if (unlikely(is_migrate_isolate_page(page))) goto out_retry; + if (unlikely(page_is_tag_storage(page))) { + migrate_tag_storage_page(page); + return 0; + } + ret = reserve_tag_storage(page, HPAGE_PMD_ORDER, GFP_HIGHUSER_MOVABLE); if (ret) goto out_retry; diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 0acef592043c..afca42ace735 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -10,8 +10,6 @@ typedef struct folio *new_folio_t(struct folio *folio, unsigned long private); typedef void free_folio_t(struct folio *folio, unsigned long private); -struct migration_target_control; - /* * Return values from addresss_space_operations.migratepage(): * - negative errno on page migration failure; @@ -57,6 +55,12 @@ struct movable_operations { void (*putback_page)(struct page *); }; +struct migration_target_control { + int nid; /* preferred node id */ + nodemask_t *nmask; + gfp_t gfp_mask; +}; + /* Defined in mm/debug.c: */ extern const char *migrate_reason_names[MR_TYPES]; diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index f37cc03f9369..c6c5c7726d26 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -29,6 +29,7 @@ enum migrate_reason { MR_CONTIG_RANGE, MR_LONGTERM_PIN, MR_DEMOTION, + MR_ARCH_1, MR_TYPES }; diff --git a/mm/internal.h b/mm/internal.h index ddf6bb6c6308..96fff5dfc041 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -949,12 +949,6 @@ static inline bool is_migrate_highatomic_page(struct page *page) void setup_zone_pageset(struct zone *zone); -struct migration_target_control { - int nid; /* preferred node id */ - nodemask_t *nmask; - gfp_t gfp_mask; -}; - /* * mm/filemap.c */ From patchwork Sun Nov 19 16:57:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460608 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 92C3FC54E76 for ; Sun, 19 Nov 2023 17:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kCKaEG2zXpaLQqzlDPz2X1luyoLx1w2BYdlRSOfvw2Y=; b=HPXqQ2qMmuUddo QEIAfAxTMvyQDgxVnKlHyJD5OOAPVlwV4ka39p6k8tHfPU+k/62MzGivl1kXRJZa78WyCrhATp1Fg n2U70RNfJqBxDdGdDjlTZtji5u+GeEoQ6ndlu20AA89AzXX0OR1E0L+2rhojUzxwkjkFszVeoq7aY 1yGuS+QUH1y4VaeNEkf6ssqP6CF53FXmwttC/tnCLXSKin/X0uNecm2fRn2EzM+HkQLFZRqRQ0t+P lRwtRw3uttC1TyUoucFAa/+yAf7Wh99jsHFrZPpk+7JzyEkz4sgPKFU8fc7rugm5reQ7CG9Qrjusc FkDv13IcJiZsX3qml1KA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l97-00ARsY-17; Sun, 19 Nov 2023 16:59:45 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l8x-00ARij-35 for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:39 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A8837FEC; Sun, 19 Nov 2023 09:00:21 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 77E993F6C4; Sun, 19 Nov 2023 08:59:30 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 22/27] arm64: mte: swap: Handle tag restoring when missing tag storage Date: Sun, 19 Nov 2023 16:57:16 +0000 Message-Id: <20231119165721.9849-23-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085936_144066_5073053F X-CRM114-Status: GOOD ( 23.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Linux restores tags when a page is swapped in and there are tags associated with the swap entry which the new page will replace. The saved tags are restored even if the page will not be mapped as tagged, to protect against cases where the page is shared between different VMAs, and is tagged in some, but untagged in others. By using this approach, the process can still access the correct tags following an mprotect(PROT_MTE) on the non-MTE enabled VMA. But this poses a challenge for managing tag storage: in the scenario above, when a new page is allocated to be swapped in for the process where it will be mapped as untagged, the corresponding tag storage block is not reserved. mte_restore_page_tags_by_swp_entry(), when it restores the saved tags, will overwrite data in the tag storage block associated with the new page, leading to data corruption if the block is in use by a process. Get around this issue by saving the tags in a new xarray, this time indexed by the page pfn, and then restoring them when tag storage is reserved for the page. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/mte_tag_storage.h | 9 ++ arch/arm64/include/asm/pgtable.h | 11 +++ arch/arm64/kernel/mte_tag_storage.c | 20 +++- arch/arm64/mm/mteswap.c | 112 +++++++++++++++++++++++ 4 files changed, 148 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/mte_tag_storage.h b/arch/arm64/include/asm/mte_tag_storage.h index 6a8b19a6a758..a3c38099fe1a 100644 --- a/arch/arm64/include/asm/mte_tag_storage.h +++ b/arch/arm64/include/asm/mte_tag_storage.h @@ -37,6 +37,15 @@ bool page_is_tag_storage(struct page *page); vm_fault_t handle_page_missing_tag_storage(struct vm_fault *vmf); vm_fault_t handle_huge_page_missing_tag_storage(struct vm_fault *vmf); + +void tags_by_pfn_lock(void); +void tags_by_pfn_unlock(void); + +void *mte_erase_tags_for_pfn(unsigned long pfn); +bool mte_save_tags_for_pfn(void *tags, unsigned long pfn); +void mte_restore_tags_for_pfn(unsigned long start_pfn, int order); + +vm_fault_t mte_try_transfer_swap_tags(swp_entry_t entry, struct page *page); #else static inline bool tag_storage_enabled(void) { diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 1704411c096d..1a25b7d601c2 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1084,6 +1084,17 @@ static inline void arch_swap_invalidate_area(int type) mte_invalidate_tags_area_by_swp_entry(type); } +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +#define __HAVE_ARCH_SWAP_PREPARE_TO_RESTORE +static inline vm_fault_t arch_swap_prepare_to_restore(swp_entry_t entry, + struct folio *folio) +{ + if (tag_storage_enabled()) + return mte_try_transfer_swap_tags(entry, &folio->page); + return 0; +} +#endif + #define __HAVE_ARCH_SWAP_RESTORE static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) { diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 5096ce859136..6b11bb408b51 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -547,8 +547,10 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp) mutex_lock(&tag_blocks_lock); /* Check again, this time with the lock held. */ - if (page_tag_storage_reserved(page)) - goto out_unlock; + if (page_tag_storage_reserved(page)) { + mutex_unlock(&tag_blocks_lock); + return 0; + } /* Make sure existing entries are not freed from out under out feet. */ xa_lock_irqsave(&tag_blocks_reserved, flags); @@ -583,9 +585,10 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp) } page_set_tag_storage_reserved(page, order); -out_unlock: mutex_unlock(&tag_blocks_lock); + mte_restore_tags_for_pfn(page_to_pfn(page), order); + return 0; out_error: @@ -612,7 +615,8 @@ void free_tag_storage(struct page *page, int order) struct tag_region *region; unsigned long page_va; unsigned long flags; - int ret; + void *tags; + int i, ret; ret = tag_storage_find_block(page, &start_block, ®ion); if (WARN_ONCE(ret, "Missing tag storage block for pfn 0x%lx", page_to_pfn(page))) @@ -622,6 +626,14 @@ void free_tag_storage(struct page *page, int order) /* Avoid writeback of dirty tag cache lines corrupting data. */ dcache_inval_tags_poc(page_va, page_va + (PAGE_SIZE << order)); + tags_by_pfn_lock(); + for (i = 0; i < (1 << order); i++) { + tags = mte_erase_tags_for_pfn(page_to_pfn(page + i)); + if (unlikely(tags)) + mte_free_tag_buf(tags); + } + tags_by_pfn_unlock(); + end_block = start_block + order_to_num_blocks(order) * region->block_size; xa_lock_irqsave(&tag_blocks_reserved, flags); diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c index 2a43746b803f..20d718a514af 100644 --- a/arch/arm64/mm/mteswap.c +++ b/arch/arm64/mm/mteswap.c @@ -20,6 +20,114 @@ void mte_free_tag_buf(void *buf) kfree(buf); } +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +static DEFINE_XARRAY(tags_by_pfn); + +void tags_by_pfn_lock(void) +{ + xa_lock(&tags_by_pfn); +} + +void tags_by_pfn_unlock(void) +{ + xa_unlock(&tags_by_pfn); +} + +void *mte_erase_tags_for_pfn(unsigned long pfn) +{ + return __xa_erase(&tags_by_pfn, pfn); +} + +bool mte_save_tags_for_pfn(void *tags, unsigned long pfn) +{ + void *entry; + int ret; + + ret = xa_reserve(&tags_by_pfn, pfn, GFP_KERNEL); + if (ret) + return true; + + tags_by_pfn_lock(); + + if (page_tag_storage_reserved(pfn_to_page(pfn))) { + tags_by_pfn_unlock(); + return false; + } + + entry = __xa_store(&tags_by_pfn, pfn, tags, GFP_ATOMIC); + if (xa_is_err(entry)) { + xa_release(&tags_by_pfn, pfn); + goto out_unlock; + } else if (entry) { + mte_free_tag_buf(entry); + } + +out_unlock: + tags_by_pfn_unlock(); + return true; +} + +void mte_restore_tags_for_pfn(unsigned long start_pfn, int order) +{ + struct page *page = pfn_to_page(start_pfn); + unsigned long pfn; + void *tags; + + tags_by_pfn_lock(); + + for (pfn = start_pfn; pfn < start_pfn + (1 << order); pfn++, page++) { + if (WARN_ON_ONCE(!page_tag_storage_reserved(page))) + continue; + + tags = mte_erase_tags_for_pfn(pfn); + if (unlikely(tags)) { + /* + * Mark the page as tagged so mte_sync_tags() doesn't + * clear the tags. + */ + WARN_ON_ONCE(!try_page_mte_tagging(page)); + mte_copy_page_tags_from_buf(page_address(page), tags); + set_page_mte_tagged(page); + mte_free_tag_buf(tags); + } + } + + tags_by_pfn_unlock(); +} + +/* + * Note on locking: swap in/out is done with the folio locked, which eliminates + * races with mte_save/restore_page_tags_by_swp_entry. + */ +vm_fault_t mte_try_transfer_swap_tags(swp_entry_t entry, struct page *page) +{ + void *swap_tags, *pfn_tags; + bool saved; + + /* + * mte_restore_page_tags_by_swp_entry() will take care of copying the + * tags over. + */ + if (likely(page_mte_tagged(page) || page_tag_storage_reserved(page))) + return 0; + + swap_tags = xa_load(&tags_by_swp_entry, entry.val); + if (!swap_tags) + return 0; + + pfn_tags = mte_allocate_tag_buf(); + if (!pfn_tags) + return VM_FAULT_OOM; + + memcpy(pfn_tags, swap_tags, MTE_PAGE_TAG_STORAGE_SIZE); + saved = mte_save_tags_for_pfn(pfn_tags, page_to_pfn(page)); + if (!saved) + mte_free_tag_buf(pfn_tags); + + return 0; +} +#endif + int mte_save_page_tags_by_swp_entry(struct page *page) { void *tags, *ret; @@ -54,6 +162,10 @@ void mte_restore_page_tags_by_swp_entry(swp_entry_t entry, struct page *page) if (!tags) return; + /* Tags will be restored when tag storage is reserved. */ + if (tag_storage_enabled() && unlikely(!page_tag_storage_reserved(page))) + return; + if (try_page_mte_tagging(page)) { mte_copy_page_tags_from_buf(page_address(page), tags); set_page_mte_tagged(page); From patchwork Sun Nov 19 16:57:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C457CC54E76 for ; Sun, 19 Nov 2023 17:00:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=UQSFW3ZYDqGCpL2DvJyBCwjrSqH5ybmSWpORDlS64iw=; b=W0db1by1XwXY7G BIhxfPYqT+Ebh/Mry2Xu6899yZ9usDcdXXjO3cKqffVXR7L4rzZ+YX2KCbDaF1weZqIAkxFGB/nMi XJLiReDtlnHwFPgzARbEVt2QXQ2vUOmKfHjRHAMDXjOOdu9aTBs8BNgfoWc2RRc0Nodd+M1xCqcKh +M6U4gQdavMo1wVJiVO4sxHffbYv6r3/elG+tcIDOutY/K4v9pXXcfJggJFkc3e7xVCeqo5jzAh0l 26BAqyOAj+6Q0bYH1jS2tr/iGYbDG/XYTW14VgPpmgNYbaN0NtoEOoudU7ocLYrXCJJXURg3YDFb0 KhrlEPkEct59OttUArcQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l9F-00AS2I-29; Sun, 19 Nov 2023 16:59:53 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l93-00ARos-1m for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:44 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E785F1480; Sun, 19 Nov 2023 09:00:26 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B408D3F6C4; Sun, 19 Nov 2023 08:59:35 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 23/27] arm64: mte: copypage: Handle tag restoring when missing tag storage Date: Sun, 19 Nov 2023 16:57:17 +0000 Message-Id: <20231119165721.9849-24-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085941_697166_E5954C0F X-CRM114-Status: GOOD ( 13.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There are several situations where copy_highpage() can end up copying tags to a page which doesn't have its tag storage reserved. One situation involves migration racing with mprotect(PROT_MTE): VMA is initially untagged, migration starts and destination page is allocated as untagged, mprotect(PROT_MTE) changes the VMA to tagged and userspace accesses the source page, thus making it tagged. The migration code then calls copy_highpage(), which will copy the tags from the source page (now tagged) to the destination page (allocated as untagged). Yes another situation can happen during THP collapse. The huge page that will replace the HPAGE_PMD_NR contiguous mapped pages is allocated with __GFP_TAGGED not set. copy_highpage() will copy the tags from the pages being replaced to the huge page which doesn't have tag storage reserved. The situation gets even more complicated when the replacement huge page is a tag storage page. The tag storage huge page will be migrated after a fault on access, but the tags from the original pages must be copied over to the huge page that will be replacing the tag storage huge page. Signed-off-by: Alexandru Elisei --- arch/arm64/mm/copypage.c | 59 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/arch/arm64/mm/copypage.c b/arch/arm64/mm/copypage.c index a7bb20055ce0..7899f38773b9 100644 --- a/arch/arm64/mm/copypage.c +++ b/arch/arm64/mm/copypage.c @@ -13,6 +13,62 @@ #include #include #include +#include + +#ifdef CONFIG_ARM64_MTE_TAG_STORAGE +static inline bool try_transfer_saved_tags(struct page *from, struct page *to) +{ + void *tags; + bool saved; + + VM_WARN_ON_ONCE(!preemptible()); + + if (page_mte_tagged(from)) { + if (likely(page_tag_storage_reserved(to))) + return false; + + tags = mte_allocate_tag_buf(); + if (WARN_ON(!tags)) + return true; + + mte_copy_page_tags_to_buf(page_address(from), tags); + saved = mte_save_tags_for_pfn(tags, page_to_pfn(to)); + if (!saved) + mte_free_tag_buf(tags); + + return saved; + } + + if (likely(!page_is_tag_storage(from))) + return false; + + tags_by_pfn_lock(); + tags = mte_erase_tags_for_pfn(page_to_pfn(from)); + tags_by_pfn_unlock(); + + if (likely(!tags)) + return false; + + if (page_tag_storage_reserved(to)) { + WARN_ON_ONCE(!try_page_mte_tagging(to)); + mte_copy_page_tags_from_buf(page_address(to), tags); + set_page_mte_tagged(to); + mte_free_tag_buf(tags); + return true; + } + + saved = mte_save_tags_for_pfn(tags, page_to_pfn(to)); + if (!saved) + mte_free_tag_buf(tags); + + return saved; +} +#else +static inline bool try_transfer_saved_tags(struct page *from, struct page *to) +{ + return false; +} +#endif void copy_highpage(struct page *to, struct page *from) { @@ -24,6 +80,9 @@ void copy_highpage(struct page *to, struct page *from) if (kasan_hw_tags_enabled()) page_kasan_tag_reset(to); + if (tag_storage_enabled() && try_transfer_saved_tags(from, to)) + return; + if (system_supports_mte() && page_mte_tagged(from)) { /* It's a new page, shouldn't have been tagged yet */ WARN_ON_ONCE(!try_page_mte_tagging(to)); From patchwork Sun Nov 19 16:57:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460610 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 746ABC54E76 for ; Sun, 19 Nov 2023 17:00:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yc1BSlIObS08b7b11J1LOkMyhh7vgILEjGoM5CSFBbw=; b=2fiH5/fsOPpl4e jJhpGrvafQqmF6x4pHIDAqifPrkLFKDZZwErPLOoLPGgHvye0KW6R8Ipbk5nDVF916G+RMixt0Ck4 iMqUfPGLwW7Rx7j1suIND+fBUBabqUl+X0AHaA+c8HRPZz33QgdacFaI+Gwlo8kmdkAi2unIOVEDh JpuEoTY94msIGtaYtTgo8tWr8kdjmVs8DqeA2/B+fcfrIVx9N+f8JICgs5BRxENpFy6oTMTGQxHN9 GotVLeh0rcJ9QXZzOnpYH07cbSWd1Pc8ov4AvhhhTm8Bt2vWdJRNtmivekfvCA6KFlxHzeHE1G9xJ jzYL/qFgHO1NDz/LbPhg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l9I-00AS5I-21; Sun, 19 Nov 2023 16:59:56 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l98-00ARtb-1Y for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:48 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2B3AD14BF; Sun, 19 Nov 2023 09:00:32 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 00E483F6C4; Sun, 19 Nov 2023 08:59:40 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 24/27] arm64: mte: Handle fatal signal in reserve_tag_storage() Date: Sun, 19 Nov 2023 16:57:18 +0000 Message-Id: <20231119165721.9849-25-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085946_640946_79C257BD X-CRM114-Status: GOOD ( 12.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org As long as a fatal signal is pending, alloc_contig_range() will fail with -EINTR. This makes it impossible for tag storage allocation to succeed, and the page allocator will print an OOM splat. The process is going to be killed, so return 0 (success) from reserve_tag_storage() to allow the page allocator to make progress. set_pte_at() will map it with PAGE_FAULT_ON_ACCESS and subsequent accesses from different threads will cause a fault until the signal is delivered. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 17 +++++++++++++++++ arch/arm64/mm/fault.c | 5 +++++ 2 files changed, 22 insertions(+) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 6b11bb408b51..602fdc70db1c 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -572,6 +572,23 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp) break; } + /* + * alloc_contig_range() returns -EINTR from + * __alloc_contig_migrate_range() if a fatal signal is pending. + * As long as the signal hasn't been handled, it is impossible + * to reserve tag storage for any page. Stop trying to reserve + * tag storage, but return 0 so the page allocator can make + * forward progress, instead of printing an OOM splat. + * + * The tagged page with missing tag storage will be mapped with + * PAGE_FAULT_ON_ACCESS in set_pte_at(), which means accesses + * until the signal is delivered will cause a fault. + */ + if (ret == -EINTR) { + ret = 0; + goto out_error; + } + if (ret) goto out_error; diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 964c5ae161a3..fdc98c5828bf 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -950,6 +950,11 @@ gfp_t arch_calc_vma_gfp(struct vm_area_struct *vma, gfp_t gfp) void tag_clear_highpage(struct page *page) { + if (tag_storage_enabled() && unlikely(!page_tag_storage_reserved(page))) { + clear_page(page_address(page)); + return; + } + /* Newly allocated page, shouldn't have been tagged yet */ WARN_ON_ONCE(!try_page_mte_tagging(page)); mte_zero_clear_page_tags(page_address(page)); From patchwork Sun Nov 19 16:57:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 04806C54E76 for ; Sun, 19 Nov 2023 17:00:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=MWf/XdiQJVv3AjL2A3m7TTdKnO2K6wor2efidYHfX0o=; b=m9SrkiqTvk3B1+ jtKCC2AEm9BEfcQ97i6YLsneqGBRkkonzlE4rj6DYW3+0NKptWKVZUbDtRQq0uBnWoGLX+9C4MAyo uoiuJwduIhtgTl11so5ABMfqSLM7KBsk5zx4XjzBkpL17ODT7HOOpkiBhy/fK33hS8gO5Rs/9D2ru WlMNQyd7LRhfBZRSXKwdTxtq34EIjFEwhJuOBkCymjhJ+SY3tX0gr4SXZXyVKG8WYEpnr7EA7R1mR OdSRF6brS/cEMTWhgzkVOQYFsAibBmjRe52XfZB26AiIVT2yYP0o77ZHDqXp/ygo+DquQjnsgG4B5 sAQVG2MEoYQuMtAV+OCQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l9S-00ASF2-18; Sun, 19 Nov 2023 17:00:06 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l9D-00ARzx-2J for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 16:59:53 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 635671007; Sun, 19 Nov 2023 09:00:37 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 36E813F6C4; Sun, 19 Nov 2023 08:59:46 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 25/27] KVM: arm64: Disable MTE if tag storage is enabled Date: Sun, 19 Nov 2023 16:57:19 +0000 Message-Id: <20231119165721.9849-26-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085951_825605_AE485D41 X-CRM114-Status: GOOD ( 11.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org KVM allows MTE enabled VMs to be created when the backing VMA does not have MTE enabled. Without changes to how KVM allocates memory for a VM, it is impossible at the moment to discern when the corresponding tag storage needs to be reserved. For now, disable MTE in KVM if tag storage is enabled. Signed-off-by: Alexandru Elisei --- arch/arm64/kvm/arm.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index e5f75f1f1085..5b33c532c62a 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -29,6 +29,7 @@ #include #include #include +#include #include #include #include @@ -86,7 +87,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, break; case KVM_CAP_ARM_MTE: mutex_lock(&kvm->lock); - if (!system_supports_mte() || kvm->created_vcpus) { + if (!system_supports_mte() || tag_storage_enabled() || + kvm->created_vcpus) { r = -EINVAL; } else { r = 0; @@ -279,7 +281,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = 1; break; case KVM_CAP_ARM_MTE: - r = system_supports_mte(); + r = system_supports_mte() && !tag_storage_enabled(); break; case KVM_CAP_STEAL_TIME: r = kvm_arm_pvtime_supported(); From patchwork Sun Nov 19 16:57:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 88A18C54E76 for ; Sun, 19 Nov 2023 18:00:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=pGw2/cgMudDLUGTgEvps2F9c1jEFDpDq2DwAYiI6X7I=; b=J2E3O7PkE9BWFW 6K62QmX4IcKLo9NAwsuFzH4ya95Pr8UsqLUhqSS5vKEGwHHn3l5ng6+wpHRzenT7OIu3QWtbNTbUP HLa7e0LAkM/dNAXrpWBfVtsy6kq1SzF3Lq5JVlclHVpaJg67TKM5dUdERLV/ODMkIXyxbJyesaTTT 9QVyku52JDeKJe1H8j1v6h7am77xrvUHT1jUvAj2augP7nu1tLiiU/xMDVCfskzcjfZ0F4xeB+iNm lGn5OQgYauCJsolcidj16jz80GePROxAz5m3Cve+8Lk7e0h97gqHZbLqcs+WFsvK3DKTo76vd9El8 gYEXS/1lIiRbzzJLHkag==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4m5m-00AVw2-0Z; Sun, 19 Nov 2023 18:00:22 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l9I-00AS5f-2i for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 17:00:02 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 929F61042; Sun, 19 Nov 2023 09:00:42 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 69E853F6C4; Sun, 19 Nov 2023 08:59:51 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 26/27] arm64: mte: Fast track reserving tag storage when the block is free Date: Sun, 19 Nov 2023 16:57:20 +0000 Message-Id: <20231119165721.9849-27-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_085957_229945_B8E8F940 X-CRM114-Status: GOOD ( 18.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org A double digit performance decrease for Chrome startup time has been reported with the dynamic tag storage management enabled. A large part of the regression is due to lru_cache_disable(), called from __alloc_contig_migrate_range(), which IPIs all CPUs in the system. Improve the performance by taking the storage block directly from the freelist if it's free, thus sidestepping the costly function call. Note that at the moment this is implemented only when the block size is 1 (the block is one page); larger block sizes could be added later if necessary. Signed-off-by: Alexandru Elisei --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/mte_tag_storage.c | 15 +++++++++++++++ include/linux/page-flags.h | 15 +++++++++++++-- mm/Kconfig | 4 ++++ mm/memory-failure.c | 8 ++++---- mm/page_alloc.c | 21 ++++++++++++--------- 6 files changed, 49 insertions(+), 15 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 3b9c435eaafb..93a4bbca3800 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2067,6 +2067,7 @@ config ARM64_MTE_TAG_STORAGE bool "Dynamic MTE tag storage management" depends on ARCH_KEEP_MEMBLOCK select ARCH_HAS_FAULT_ON_ACCESS + select WANTS_TAKE_PAGE_OFF_BUDDY select CONFIG_CMA help Adds support for dynamic management of the memory used by the hardware diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 602fdc70db1c..11961587382d 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -522,6 +522,7 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp) unsigned long block; unsigned long flags; unsigned int tries; + bool success; int ret = 0; VM_WARN_ON_ONCE(!preemptible()); @@ -565,6 +566,19 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp) if (tag_storage_block_is_reserved(block)) continue; + if (region->block_size == 1 && is_free_buddy_page(pfn_to_page(block))) { + success = take_page_off_buddy(pfn_to_page(block), false); + if (success) { + ret = tag_storage_reserve_block(block, region, order); + if (ret) { + put_page_back_buddy(pfn_to_page(block), false); + goto out_error; + } + page_ref_inc(pfn_to_page(block)); + goto success_next; + } + } + tries = 3; while (tries--) { ret = alloc_contig_range(block, block + region->block_size, MIGRATE_CMA, gfp); @@ -598,6 +612,7 @@ int reserve_tag_storage(struct page *page, int order, gfp_t gfp) goto out_error; } +success_next: count_vm_events(CMA_ALLOC_SUCCESS, region->block_size); } diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 7915165a51bd..0d0380141f5d 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -576,11 +576,22 @@ TESTSCFLAG(HWPoison, hwpoison, PF_ANY) #define MAGIC_HWPOISON 0x48575053U /* HWPS */ extern void SetPageHWPoisonTakenOff(struct page *page); extern void ClearPageHWPoisonTakenOff(struct page *page); -extern bool take_page_off_buddy(struct page *page); -extern bool put_page_back_buddy(struct page *page); +extern bool PageHWPoisonTakenOff(struct page *page); #else PAGEFLAG_FALSE(HWPoison, hwpoison) +TESTSCFLAG_FALSE(HWPoison, hwpoison) #define __PG_HWPOISON 0 +static inline void SetPageHWPoisonTakenOff(struct page *page) { } +static inline void ClearPageHWPoisonTakenOff(struct page *page) { } +static inline bool PageHWPoisonTakenOff(struct page *page) +{ + return false; +} +#endif + +#ifdef CONFIG_WANTS_TAKE_PAGE_OFF_BUDDY +extern bool take_page_off_buddy(struct page *page, bool poison); +extern bool put_page_back_buddy(struct page *page, bool unpoison); #endif #if defined(CONFIG_PAGE_IDLE_FLAG) && defined(CONFIG_64BIT) diff --git a/mm/Kconfig b/mm/Kconfig index a90eefc3ee80..0766cdc3de4d 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -773,6 +773,7 @@ config MEMORY_FAILURE depends on MMU depends on ARCH_SUPPORTS_MEMORY_FAILURE bool "Enable recovery from hardware memory errors" + select WANTS_TAKE_PAGE_OFF_BUDDY select MEMORY_ISOLATION select RAS help @@ -1022,6 +1023,9 @@ config ARCH_HAS_CACHE_LINE_SIZE config ARCH_HAS_FAULT_ON_ACCESS bool +config WANTS_TAKE_PAGE_OFF_BUDDY + bool + config ARCH_HAS_CURRENT_STACK_POINTER bool help diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 660c21859118..8b44afd6a558 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -157,7 +157,7 @@ static int __page_handle_poison(struct page *page) zone_pcp_disable(page_zone(page)); ret = dissolve_free_huge_page(page); if (!ret) - ret = take_page_off_buddy(page); + ret = take_page_off_buddy(page, true); zone_pcp_enable(page_zone(page)); return ret; @@ -1348,7 +1348,7 @@ static int page_action(struct page_state *ps, struct page *p, return action_result(pfn, ps->type, result); } -static inline bool PageHWPoisonTakenOff(struct page *page) +bool PageHWPoisonTakenOff(struct page *page) { return PageHWPoison(page) && page_private(page) == MAGIC_HWPOISON; } @@ -2236,7 +2236,7 @@ int memory_failure(unsigned long pfn, int flags) res = get_hwpoison_page(p, flags); if (!res) { if (is_free_buddy_page(p)) { - if (take_page_off_buddy(p)) { + if (take_page_off_buddy(p, true)) { page_ref_inc(p); res = MF_RECOVERED; } else { @@ -2567,7 +2567,7 @@ int unpoison_memory(unsigned long pfn) ret = folio_test_clear_hwpoison(folio) ? 0 : -EBUSY; } else if (ghp < 0) { if (ghp == -EHWPOISON) { - ret = put_page_back_buddy(p) ? 0 : -EBUSY; + ret = put_page_back_buddy(p, true) ? 0 : -EBUSY; } else { ret = ghp; unpoison_pr_info("Unpoison: failed to grab page %#lx\n", diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 135f9283a863..4b74acfc41a6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6700,7 +6700,7 @@ bool is_free_buddy_page(struct page *page) } EXPORT_SYMBOL(is_free_buddy_page); -#ifdef CONFIG_MEMORY_FAILURE +#ifdef CONFIG_WANTS_TAKE_PAGE_OFF_BUDDY /* * Break down a higher-order page in sub-pages, and keep our target out of * buddy allocator. @@ -6730,11 +6730,10 @@ static void break_down_buddy_pages(struct zone *zone, struct page *page, set_buddy_order(current_buddy, high); } } - /* - * Take a page that will be marked as poisoned off the buddy allocator. + * Take a page off the buddy allocator, and optionally mark it as poisoned. */ -bool take_page_off_buddy(struct page *page) +bool take_page_off_buddy(struct page *page, bool poison) { struct zone *zone = page_zone(page); unsigned long pfn = page_to_pfn(page); @@ -6755,7 +6754,8 @@ bool take_page_off_buddy(struct page *page) del_page_from_free_list(page_head, zone, page_order); break_down_buddy_pages(zone, page_head, page, 0, page_order, migratetype); - SetPageHWPoisonTakenOff(page); + if (poison) + SetPageHWPoisonTakenOff(page); if (!is_migrate_isolate(migratetype)) __mod_zone_freepage_state(zone, -1, migratetype); ret = true; @@ -6769,9 +6769,10 @@ bool take_page_off_buddy(struct page *page) } /* - * Cancel takeoff done by take_page_off_buddy(). + * Cancel takeoff done by take_page_off_buddy(), and optionally unpoison the + * page. */ -bool put_page_back_buddy(struct page *page) +bool put_page_back_buddy(struct page *page, bool unpoison) { struct zone *zone = page_zone(page); unsigned long pfn = page_to_pfn(page); @@ -6781,9 +6782,11 @@ bool put_page_back_buddy(struct page *page) spin_lock_irqsave(&zone->lock, flags); if (put_page_testzero(page)) { - ClearPageHWPoisonTakenOff(page); + VM_WARN_ON_ONCE(PageHWPoisonTakenOff(page) && !unpoison); + if (unpoison) + ClearPageHWPoisonTakenOff(page); __free_one_page(page, pfn, zone, 0, migratetype, FPI_NONE); - if (TestClearPageHWPoison(page)) { + if (!unpoison || (unpoison && TestClearPageHWPoison(page))) { ret = true; } } From patchwork Sun Nov 19 16:57:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 13460612 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5497CC54E76 for ; Sun, 19 Nov 2023 17:00:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=f42na97B1IaUCP9C4mwp2AiYRsMGERLSIyo/7fhT/Vo=; b=YSbvDloKoQ4GJD UXtXs+PuCcsgrV+IiBWgb4B+E2NiW1iD1dh8FL14RU4GcXTgPqaCYQcVCN9j1QM90chW8RNAPjwXk gbNbeuVxY/PSvcTzkGYdJHhaaeJmwwckpYlmiQ5GuA+rCutuQqD0c6PLF5QmoZ8YQDbu66ZS0hWPx JqUopBT6DCUIwOfmvsVNebHXK6hvsnAgoORzvNXJ48gKffE9AHWCKXa4aup+XB22MHyFj+7/L1wjU HsEfyEfhqWf58lLZtzdOXJ4l/HFPdWRP3eHcOpsOeshqPCmFJyqwFOOwBXGH8kW6XSKhujhfPniPz vnXmsx0OrjdumTN3D+og==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l9f-00ASR8-1l; Sun, 19 Nov 2023 17:00:19 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r4l9O-00ASB4-0f for linux-arm-kernel@lists.infradead.org; Sun, 19 Nov 2023 17:00:04 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C6B171476; Sun, 19 Nov 2023 09:00:47 -0800 (PST) Received: from e121798.cable.virginm.net (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9BFC23F6C4; Sun, 19 Nov 2023 08:59:56 -0800 (PST) From: Alexandru Elisei To: catalin.marinas@arm.com, will@kernel.org, oliver.upton@linux.dev, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, arnd@arndb.de, akpm@linux-foundation.org, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, mhiramat@kernel.org, rppt@kernel.org, hughd@google.com Cc: pcc@google.com, steven.price@arm.com, anshuman.khandual@arm.com, vincenzo.frascino@arm.com, david@redhat.com, eugenis@google.com, kcc@google.com, hyesoo.yu@samsung.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, linux-fsdevel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org Subject: [PATCH RFC v2 27/27] arm64: mte: Enable dynamic tag storage reuse Date: Sun, 19 Nov 2023 16:57:21 +0000 Message-Id: <20231119165721.9849-28-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231119165721.9849-1-alexandru.elisei@arm.com> References: <20231119165721.9849-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231119_090002_409861_B7ED1774 X-CRM114-Status: UNSURE ( 7.51 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Everything is in place, enable tag storage management. Signed-off-by: Alexandru Elisei --- arch/arm64/kernel/mte_tag_storage.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/arm64/kernel/mte_tag_storage.c b/arch/arm64/kernel/mte_tag_storage.c index 11961587382d..9f60e952a814 100644 --- a/arch/arm64/kernel/mte_tag_storage.c +++ b/arch/arm64/kernel/mte_tag_storage.c @@ -395,6 +395,9 @@ static int __init mte_tag_storage_activate_regions(void) reserve_tag_storage(ZERO_PAGE(0), 0, GFP_HIGHUSER_MOVABLE); + static_branch_enable(&tag_storage_enabled_key); + pr_info("MTE tag storage region management enabled"); + return 0; out_disabled: