From patchwork Sat Jan 13 09:44:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nanyong Sun X-Patchwork-Id: 13518906 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3168C4707C for ; Sat, 13 Jan 2024 08:48:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 780596B0092; Sat, 13 Jan 2024 03:48:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F7CF8D0005; Sat, 13 Jan 2024 03:48:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 260838D0003; Sat, 13 Jan 2024 03:48:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F00026B0092 for ; Sat, 13 Jan 2024 03:48:22 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B7E02C0674 for ; Sat, 13 Jan 2024 08:48:22 +0000 (UTC) X-FDA: 81673661244.07.E2D51A5 Received: from szxga08-in.huawei.com (szxga08-in.huawei.com [45.249.212.255]) by imf01.hostedemail.com (Postfix) with ESMTP id 2CF694000D for ; Sat, 13 Jan 2024 08:48:19 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf01.hostedemail.com: domain of sunnanyong@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=sunnanyong@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705135701; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dvLlTM0WXVWEZhImNM4s3rHk6rhpAi103dlOK2FXokA=; b=hNKs7/v/hrBf35aqFInLhw+3CIM7QdbYrFY1jJy4i9JXR0yP9aTS/wB/9l16I+asgwUBlo xgIHrps2eK9bRON/9rVd7JZ9xiuzRYrD2piQXbFUIYXcGvqawGAXo3KSHqcoyO3m+XZjkE BogO/hO4kgA/L13PL4VT6S4futoYdDQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf01.hostedemail.com: domain of sunnanyong@huawei.com designates 45.249.212.255 as permitted sender) smtp.mailfrom=sunnanyong@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705135701; a=rsa-sha256; cv=none; b=IolaHmOtAeX4F8DGHGCNF28PPNcp9y2VjxUH2eJW+WQcfa13cqgyPRBgZ0oYnAYkHOBIc1 96bfMwON67sJ5AU02BZ/U5N43SmYTSWfsgzaLywxNoVuQZktcIuAk8xjc9R2Sw7gk7+MZ2 rgpNcqpauX1ANRu1LZxaQr5ZI643mfo= Received: from mail.maildlp.com (unknown [172.19.88.194]) by szxga08-in.huawei.com (SkyGuard) with ESMTP id 4TBsV64Wj5z1Q7CJ; Sat, 13 Jan 2024 16:47:26 +0800 (CST) Received: from kwepemm600003.china.huawei.com (unknown [7.193.23.202]) by mail.maildlp.com (Postfix) with ESMTPS id C49BF1402E2; Sat, 13 Jan 2024 16:48:14 +0800 (CST) Received: from huawei.com (10.175.113.32) by kwepemm600003.china.huawei.com (7.193.23.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Sat, 13 Jan 2024 16:48:13 +0800 From: Nanyong Sun To: , , , , , CC: , , , , , Subject: [PATCH v3 1/3] mm: HVO: introduce helper function to update and flush pgtable Date: Sat, 13 Jan 2024 17:44:34 +0800 Message-ID: <20240113094436.2506396-2-sunnanyong@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240113094436.2506396-1-sunnanyong@huawei.com> References: <20240113094436.2506396-1-sunnanyong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.32] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemm600003.china.huawei.com (7.193.23.202) X-Rspamd-Queue-Id: 2CF694000D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: q7axzb87cyk5qn75nmw54d6ucmf1y76z X-HE-Tag: 1705135699-757117 X-HE-Meta: U2FsdGVkX18Ul13JCuWJLmvTER7DzLbiIDLkpjWwQapLgyEgEsa7wL+8reKYDiDY4zKfh64ul4w7WdfMTZvE3IhZrikB42zXG4de5KrLdM3Di2QqNbGANys2G+FRTnar0JVdQ4+WqHCUABAd0RWxi5tbQrh5yOlbdo03KUZ+dkuWVa3eOkDbREIwNp5QpXAfHa4cZD2BRedQye+qn4GLA+ccVfeQpKVhtv9BIoQiUFwwWjGGBTPD/ujxPbcxTom1a1QeOveAHf0d3jbq+AVyGveuKetamaikmTB7p2DG9CFZB64DYlKZ06jPw6x0SBmDUgxf5p4CZ7FUM5ihe6S7Kzmj2lAO5w7NfQd+PkejrSi3wMVTs8xxLOKnmEeDIwyZqQmajAM7nza+cObsW4QrhhVm46ncivr8DOCgfW88p61+dioMl3DFpoksGMnis7C0SK3rjOIoQArxHMepfgGaIBgvzVp2Uje2p5LnSa1Ty9R6twIQMpzN/Xp6G8zvK3IdRukEA1kN3kRUsgOAax8NpppOPLQ7EqSXLCyAIOOqsraIihaRN1CgKlQ7upA1VfLr/4IKambVSgTehDcxewo6N7mKYpCwEyza77g/PdnpgzlWiml7f9NomwiPXZ/+RdIHPOzl2qP/FgpsGFrSQpMowNAXbN95EjOGAUD0f6RV66iGk/Ewh/BAb0r7j1j+HvbhjWZGosGt/9TdJKi3o2bkWeWaDCBqNTN8dbaLW8/U6+oZIAg1qWsO8IhhAB8gnXb//OZH8Pw9U2YDLhXWkHtdcOQok7RKtCJI1gQvOgDpiH+LWYcApEqJcUVOLV0Je1lGiYY0h6bNnguskcyLC6DAe8xQgHD63TFf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add pmd/pte update and tlb flush helper function to update page table. This refactoring patch is designed to facilitate each architecture to implement its own special logic in preparation for the arm64 architecture to follow the necessary break-before-make sequence when updating page tables. Signed-off-by: Nanyong Sun Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 55 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index da177e49d956..f1f5702bce4f 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -46,6 +46,37 @@ struct vmemmap_remap_walk { unsigned long flags; }; +#ifndef vmemmap_update_pmd +static inline void vmemmap_update_pmd(unsigned long addr, + pmd_t *pmdp, pte_t *ptep) +{ + pmd_populate_kernel(&init_mm, pmdp, ptep); +} +#endif + +#ifndef vmemmap_update_pte +static inline void vmemmap_update_pte(unsigned long addr, + pte_t *ptep, pte_t pte) +{ + set_pte_at(&init_mm, addr, ptep, pte); +} +#endif + +#ifndef vmemmap_flush_tlb_all +static inline void vmemmap_flush_tlb_all(void) +{ + flush_tlb_all(); +} +#endif + +#ifndef vmemmap_flush_tlb_range +static inline void vmemmap_flush_tlb_range(unsigned long start, + unsigned long end) +{ + flush_tlb_kernel_range(start, end); +} +#endif + static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, struct vmemmap_remap_walk *walk) { @@ -81,9 +112,9 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start, /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); - pmd_populate_kernel(&init_mm, pmd, pgtable); + vmemmap_update_pmd(start, pmd, pgtable); if (!(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)) - flush_tlb_kernel_range(start, start + PMD_SIZE); + vmemmap_flush_tlb_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -171,7 +202,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) - flush_tlb_kernel_range(start, end); + vmemmap_flush_tlb_range(start, end); return 0; } @@ -217,15 +248,15 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, /* * Makes sure that preceding stores to the page contents from - * vmemmap_remap_free() become visible before the set_pte_at() - * write. + * vmemmap_remap_free() become visible before the + * vmemmap_update_pte() write. */ smp_wmb(); } entry = mk_pte(walk->reuse_page, pgprot); list_add(&page->lru, walk->vmemmap_pages); - set_pte_at(&init_mm, addr, pte, entry); + vmemmap_update_pte(addr, pte, entry); } /* @@ -264,10 +295,10 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, /* * Makes sure that preceding stores to the page contents become visible - * before the set_pte_at() write. + * before the vmemmap_update_pte() write. */ smp_wmb(); - set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); + vmemmap_update_pte(addr, pte, mk_pte(page, pgprot)); } /** @@ -519,7 +550,7 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, } if (restored) - flush_tlb_all(); + vmemmap_flush_tlb_all(); if (!ret) ret = restored; return ret; @@ -642,7 +673,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l break; } - flush_tlb_all(); + vmemmap_flush_tlb_all(); list_for_each_entry(folio, folio_list, lru) { int ret; @@ -659,7 +690,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l * allowing more vmemmap remaps to occur. */ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { - flush_tlb_all(); + vmemmap_flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); __hugetlb_vmemmap_optimize_folio(h, folio, &vmemmap_pages, @@ -667,7 +698,7 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l } } - flush_tlb_all(); + vmemmap_flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } From patchwork Sat Jan 13 09:44:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nanyong Sun X-Patchwork-Id: 13518905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6AF5C4706C for ; Sat, 13 Jan 2024 08:48:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 409EE8D0001; Sat, 13 Jan 2024 03:48:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 373D16B0093; Sat, 13 Jan 2024 03:48:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 105158D0001; Sat, 13 Jan 2024 03:48:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E3F498D0003 for ; Sat, 13 Jan 2024 03:48:22 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B87DD802B9 for ; Sat, 13 Jan 2024 08:48:22 +0000 (UTC) X-FDA: 81673661244.08.62170FA Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by imf07.hostedemail.com (Postfix) with ESMTP id 296554000C for ; Sat, 13 Jan 2024 08:48:19 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf07.hostedemail.com: domain of sunnanyong@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=sunnanyong@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705135701; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JHEdtIrlNv7Ds/Ua+BhdGd9azXnr4YIYBkLQe6JULI0=; b=XegG23Mf1UEe4N06nzQt38/HXc/hJEFnZejlWDevjn1g8BdSojMfd1W6ShrfYydAXZG/mW 03TxTssGtm18HWqxZp5vsfSN8Kp4wnhofO32fJTRvXrEhnjc5RI0e/lHg8gcMY1KxrGOk7 eXxRjjN1NgrmfLt0kWdLN4AXQJ1icjM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf07.hostedemail.com: domain of sunnanyong@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=sunnanyong@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705135701; a=rsa-sha256; cv=none; b=eR0cFlK+atbkQ30J7ysBky1qkpZDz1xRc9mfpK2AHjYHRCZAD3tqGElq1sZ/l7Y3S/vfY2 j5qjZM3DoCWKvSgIk+91/ykELDL2yBHXk6hp53BCQtMpi3g9TNa+kLln8aX8+8fhVhQ3+p 4a26a6Tbjz6eHdwEBzGyNN6+eekfH+c= Received: from mail.maildlp.com (unknown [172.19.162.254]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4TBsVG5rYczNl3S; Sat, 13 Jan 2024 16:47:34 +0800 (CST) Received: from kwepemm600003.china.huawei.com (unknown [7.193.23.202]) by mail.maildlp.com (Postfix) with ESMTPS id AF13118001D; Sat, 13 Jan 2024 16:48:15 +0800 (CST) Received: from huawei.com (10.175.113.32) by kwepemm600003.china.huawei.com (7.193.23.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Sat, 13 Jan 2024 16:48:14 +0800 From: Nanyong Sun To: , , , , , CC: , , , , , Subject: [PATCH v3 2/3] arm64: mm: HVO: support BBM of vmemmap pgtable safely Date: Sat, 13 Jan 2024 17:44:35 +0800 Message-ID: <20240113094436.2506396-3-sunnanyong@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240113094436.2506396-1-sunnanyong@huawei.com> References: <20240113094436.2506396-1-sunnanyong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.32] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemm600003.china.huawei.com (7.193.23.202) X-Rspam-User: X-Stat-Signature: cb41crkd8ieczhp1boc475am7ixjpruj X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 296554000C X-HE-Tag: 1705135699-928252 X-HE-Meta: U2FsdGVkX1+wste2m4PAAiNdJ8scgvCZr4qOgCirZ4mSUFzNbb3ga5EOfYf1MUlG9L9mrJG4Eh04itoku2EKSE1r2APob7lRh9hrYTzX76bfC9aqOoOI7CJZuSyJuGmqzYlcoXzgeQCLtuA9Ov3Cb4RIkbk1/bSnLt7cZbDpTDmDNaFPUV0aFLYn9+7MGTu5dohZglS9uvjcSz51aDd92fiAndY4OGUpYZIkPWYBEkoX+4Hy1AO8ZJ97jORHqN96ksMWt2j5squnzmMm0pieE6+ao6QgxajH1S0yyn4qZpRTOrq0RJ/tLgSiODJNbHPu9UpjtYa8q5x9O8IM72yihwAbX7uLfjobqz8osvwoM1wQTtUbkEc1ZxuVYoLtlH1WDetQqrNZAfn96eafVHmHfhOLN9T+hCJhYJsRhHRDPmVh2IkJc5TeMrXX3ESKA8nCP18b7i8ocAiKGzZNuSPzCATqCq5iDqTD7T8rn9mRPukx8WbSEK/fsjU+QasLGFAYht+yTWbK/25kg+qGz/tcOcKiHPwUNDAksrZf7Yw63MS69Ju7pwDLKVpMJIOdSC5mKoE3UiNCjgrHLBiLqFfzAFEqH4Ih+dJNfhHH3hqx5FdC8/6668yQis/G7tAGpPhhjqgxa6Gfdl2yqvgSTgAEL2Ha17uDZy2dUbcual3brJP1ikpsWMav6R+M05nWh0LudQyzMvEXQYWqfYob1c5XDHgbL73Tp/yQQKvmSKK6g4h4kEGWOem0dVrXAqK0TjmGEbMOLuzS1KgXJueRcZDky9WjHeeQqbju0ZMsHtDfgWMPTnxeVQRdWc89jFiMyryNBurAOcOrq7WTIvEjgMmeBnXYeZWpyhaOmVPjWApaq0ilWJo+A4/2JURYjKpU+T2zvTVoDJvS6pgkFqR88DUPbnwYnbmj8kVMCUoxh8OQ605c6h9iyAjqsYRr3JY9M++LqOR09Lwifjs63lh1iBa NsgeOP1v xhuLTR8BjwrIX+VchwcV6zz1jzXNjt2z0FaD8oneZ3pUzZuJNgsDEnAjm37Hk+uQE38iWzWojVYWffe7j5nl7ht7S7nITI2ozPjq+NxLW0tWBpZi3FKAfB5Yu0wFxJbguyrFCkAz4WlQ/IP5daiEW3Wtqvg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Implement vmemmap_update_pmd and vmemmap_update_pte on arm64 to do BBM(break-before-make) logic when change the page table of vmemmap address, they will under the init_mm.page_table_lock. If a translation fault of vmemmap address concurrently happened after pte/pmd cleared, vmemmap page fault handler will acquire the init_mm.page_table_lock to wait for vmemmap update to complete, by then the virtual address is valid again, so PF can return and access can continue. In other case, do the traditional kernel fault. Implement vmemmap_flush_tlb_all/range on arm64 with nothing to do because tlb already flushed in every single BBM. Signed-off-by: Nanyong Sun Reviewed-by: Muchun Song --- arch/arm64/include/asm/esr.h | 4 ++ arch/arm64/include/asm/pgtable.h | 8 ++++ arch/arm64/include/asm/tlbflush.h | 16 +++++++ arch/arm64/mm/fault.c | 78 +++++++++++++++++++++++++++++-- arch/arm64/mm/mmu.c | 28 +++++++++++ 5 files changed, 131 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index ae35939f395b..1c63256efd25 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -116,6 +116,10 @@ #define ESR_ELx_FSC_SERROR (0x11) #define ESR_ELx_FSC_ACCESS (0x08) #define ESR_ELx_FSC_FAULT (0x04) +#define ESR_ELx_FSC_FAULT_L0 (0x04) +#define ESR_ELx_FSC_FAULT_L1 (0x05) +#define ESR_ELx_FSC_FAULT_L2 (0x06) +#define ESR_ELx_FSC_FAULT_L3 (0x07) #define ESR_ELx_FSC_PERM (0x0C) #define ESR_ELx_FSC_SEA_TTW0 (0x14) #define ESR_ELx_FSC_SEA_TTW1 (0x15) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 79ce70fbb751..b50270107e2f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1124,6 +1124,14 @@ extern pte_t ptep_modify_prot_start(struct vm_area_struct *vma, extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +void vmemmap_update_pmd(unsigned long addr, pmd_t *pmdp, pte_t *ptep); +#define vmemmap_update_pmd vmemmap_update_pmd +void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); +#define vmemmap_update_pte vmemmap_update_pte +#endif + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 1deb5d789c2e..79e932a1bdf8 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -504,6 +504,22 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr) dsb(ish); isb(); } + +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static inline void vmemmap_flush_tlb_all(void) +{ + /* do nothing, already flushed tlb in every single BBM */ +} +#define vmemmap_flush_tlb_all vmemmap_flush_tlb_all + +static inline void vmemmap_flush_tlb_range(unsigned long start, + unsigned long end) +{ + /* do nothing, already flushed tlb in every single BBM */ +} +#define vmemmap_flush_tlb_range vmemmap_flush_tlb_range +#endif + #endif #endif diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 55f6455a8284..13189322a38f 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -368,6 +368,75 @@ static bool is_el1_mte_sync_tag_check_fault(unsigned long esr) return false; } +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +static inline bool vmemmap_fault_may_fixup(unsigned long addr, + unsigned long esr) +{ + if (addr < VMEMMAP_START || addr >= VMEMMAP_END) + return false; + + /* + * Only try to handle translation fault level 2 or level 3, + * because hugetlb vmemmap optimize only clear pmd or pte. + */ + switch (esr & ESR_ELx_FSC) { + case ESR_ELx_FSC_FAULT_L2: + case ESR_ELx_FSC_FAULT_L3: + return true; + default: + return false; + } +} + +/* + * PMD mapped vmemmap should has been split as PTE mapped + * by HVO now, here we only check this case, other cases + * should fail. + * Also should check the addr is healthy enough that will not cause + * a level2 or level3 translation fault again after page fault + * handled with success, so we need check both bits[1:0] of PMD and + * PTE as ARM Spec mentioned below: + * A Translation fault is generated if bits[1:0] of a translation + * table descriptor identify the descriptor as either a Fault + * encoding or a reserved encoding. + */ +static inline bool vmemmap_addr_healthy(unsigned long addr) +{ + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + pmdp = pmd_off_k(addr); + pmd = pmdp_get(pmdp); + if (!pmd_table(pmd)) + return false; + + ptep = pte_offset_kernel(pmdp, addr); + pte = ptep_get(ptep); + return (pte_val(pte) & PTE_TYPE_MASK) == PTE_TYPE_PAGE; +} + +static bool vmemmap_handle_page_fault(unsigned long addr, + unsigned long esr) +{ + bool ret; + + if (likely(!vmemmap_fault_may_fixup(addr, esr))) + return false; + + spin_lock(&init_mm.page_table_lock); + ret = vmemmap_addr_healthy(addr); + spin_unlock(&init_mm.page_table_lock); + + return ret; +} +#else +static inline bool vmemmap_handle_page_fault(unsigned long addr, + unsigned long esr) +{ + return false; +} +#endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ + static bool is_translation_fault(unsigned long esr) { return (esr & ESR_ELx_FSC_TYPE) == ESR_ELx_FSC_FAULT; @@ -405,9 +474,12 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, } else if (addr < PAGE_SIZE) { msg = "NULL pointer dereference"; } else { - if (is_translation_fault(esr) && - kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs)) - return; + if (is_translation_fault(esr)) { + if (kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs)) + return; + if (vmemmap_handle_page_fault(addr, esr)) + return; + } msg = "paging request"; } diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 1ac7467d34c9..d794b2f4b5a3 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1146,6 +1146,34 @@ int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, return 1; } +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP +/* + * In the window between the page table entry is cleared and filled + * with a new value, other threads have the opportunity to concurrently + * access the vmemmap area then page translation fault occur. + * Therefore, we need to ensure that the init_mm.page_table_lock is held + * to synchronize the vmemmap page fault handling which will wait for + * this lock to be released to ensure that the page table entry has been + * refreshed with a new valid value. + */ +void vmemmap_update_pmd(unsigned long addr, pmd_t *pmdp, pte_t *ptep) +{ + lockdep_assert_held(&init_mm.page_table_lock); + pmd_clear(pmdp); + flush_tlb_kernel_range(addr, addr + PMD_SIZE); + pmd_populate_kernel(&init_mm, pmdp, ptep); +} + +void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte) +{ + spin_lock(&init_mm.page_table_lock); + pte_clear(&init_mm, addr, ptep); + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); + set_pte_at(&init_mm, addr, ptep, pte); + spin_unlock(&init_mm.page_table_lock); +} +#endif + int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { From patchwork Sat Jan 13 09:44:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nanyong Sun X-Patchwork-Id: 13518904 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9FB31C4707C for ; Sat, 13 Jan 2024 08:48:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 106558D0006; Sat, 13 Jan 2024 03:48:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 01FD78D0005; Sat, 13 Jan 2024 03:48:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DABCB8D0001; Sat, 13 Jan 2024 03:48:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CB3826B0092 for ; Sat, 13 Jan 2024 03:48:22 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 9A438402C7 for ; Sat, 13 Jan 2024 08:48:22 +0000 (UTC) X-FDA: 81673661244.07.94CCCD4 Received: from szxga04-in.huawei.com (szxga04-in.huawei.com [45.249.212.190]) by imf28.hostedemail.com (Postfix) with ESMTP id 2FF7FC0008 for ; Sat, 13 Jan 2024 08:48:19 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf28.hostedemail.com: domain of sunnanyong@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=sunnanyong@huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705135700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3eklEW/rS0FJ7aXyEaabkKXSabEhwsZJbeZifNjfxro=; b=UjGUlma/RLvPlf3ibuehT5lwWYoiAOlksCMTR4OX0IwTIVcjRihW8x/Efenx46hSNhxDCc xbai2mFHi7X8pmkMR7OsIO9AGL5qJJhru63mmnQ3LXps0Oli5PWVwPA+WuNfsEwMwmX/Te bMfzieqFkwyRCpIxhqEOnkjcvFey6tk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf28.hostedemail.com: domain of sunnanyong@huawei.com designates 45.249.212.190 as permitted sender) smtp.mailfrom=sunnanyong@huawei.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705135700; a=rsa-sha256; cv=none; b=tXqWZgPDkjHbeVA8TnNr1QPdwoQg0Xmm31z+duXup+db33XuuX106mwsm89QcVtAZaI/nG 3wsWmQlXLUZJzaVcsc/2jmDsgCGrUUu0m0YHh03nQyDfk2zOb9xoMnB8GmLGHktHNyIbA9 YZ4fpS6KWtgSEI0Sye+1MEDILwVQBao= Received: from mail.maildlp.com (unknown [172.19.163.44]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4TBsTG0PbQz29jqm; Sat, 13 Jan 2024 16:46:42 +0800 (CST) Received: from kwepemm600003.china.huawei.com (unknown [7.193.23.202]) by mail.maildlp.com (Postfix) with ESMTPS id 8FEF214011A; Sat, 13 Jan 2024 16:48:16 +0800 (CST) Received: from huawei.com (10.175.113.32) by kwepemm600003.china.huawei.com (7.193.23.202) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Sat, 13 Jan 2024 16:48:15 +0800 From: Nanyong Sun To: , , , , , CC: , , , , , Subject: [PATCH v3 3/3] arm64: mm: Re-enable OPTIMIZE_HUGETLB_VMEMMAP Date: Sat, 13 Jan 2024 17:44:36 +0800 Message-ID: <20240113094436.2506396-4-sunnanyong@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240113094436.2506396-1-sunnanyong@huawei.com> References: <20240113094436.2506396-1-sunnanyong@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.175.113.32] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemm600003.china.huawei.com (7.193.23.202) X-Rspamd-Queue-Id: 2FF7FC0008 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: afk77a5zswao3hnjbj48iubcjodnah9r X-HE-Tag: 1705135699-807213 X-HE-Meta: U2FsdGVkX1+A3Z2NQ7/RW2xk6cn1lb/eUxJYQ/ozq1Yo+wjcIGvfspxAfw10tK6B161ZGfoireo4RFqA8ahzAaR79WoL6upuBVdVxGs3orPetS1qLG886+UWFVzyvMjiYT36p15ZEaoPdwY3qhKyDGj7NwxXX4NX/wXArX6wWGpQhlHKPIim6ZrPM0+3ub0WFPylW7nWxbzQIwbozgKZisFSjj3+9PpFQ5YJguUZMJGb6ZyTHisbg31830HMAvJII+MKAIKodbbQvAWFm9UlS8zos+W/KohNNkNZU97/iilAz3LgeVBAMFMWrdAQ6KiRXSlBZeXlLkrjD+mDeG6WBn3tW+xLBoX6H5Dvf/y1KvQjhJ9aZT8i6e4yoLUoT7XcXFAg/sWgXEjhxiVo2v4vnsLoyZjVIKP3sumqWi2qr+9GnNQBfbKDtSxePKaR3cM+opb2/+QvU0ja8Vvbp67i40INqx23y4+33raNTg57eK/atLQwB+piMZeTidul7dLCEe7GjCE8ocMstz/96Qm8riX3KF/t74j7vaPQGNJEGMxrywWHQtc2OqLcd7wfSps7MzlgpNbAmE3w90q1IXLCyR3XKJSO4GhaUeygbAuRi9p+liVC7psmadHkEn/diphfhgxEiUGHbxhW92o8K8MZYeO+ws7m0Goay1ZwHFol+AanMH+Mp/ZBVbSqStDUIGuEtvggvv3PgQG3yRKXVsc+V0otV/HxTaYH+n6FbK8otUHOt9VY/LSB0xnmz/wlWWvV3S7WGAlwlb67bj2svFLSXkN9qT11SdmOaUMruJwT/J0Nalqo9CPrFsRcx3veDF8BbTgegYIbX8enVk35SMyeFHA2vkOICClHqeetkHh7yFmytFmpfDeU6W/nBCEpvlaa+NuMAeq+GfKED54uIg+rypNRGV02mHiS+8RD3w0nmUln41wSxYoVNl6zwAKYuXusU059JPC4+EAvODX/5zv YaeiEH/X kSSW9/C7PlcsiHDclD8yS0DEcpxsS9tIjrRcp2QC72R8573miFsdCm5kll50w7TJo2VBmMIgBQIj2+k17hgoKKZm215oMLNi79ZsRBR3hWBxu0scS6xlqcr0p4HYpZaluu+N4gk38QXa8YxH4g4mfwHel/Dg4f1Xe5CsY8TQIWClZr0iF18hKPMfsoyyX2G4az3MVeHZvMFEkWkMQwxkxKQVOAJVgkKS6o6qwah1q0lFadtOBsUKnHC8ZWSnhrdxmjfjX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now update of vmemmap page table can follow the rule of break-before-make safely for arm64 architecture, re-enable HVO on arm64. Signed-off-by: Nanyong Sun Reviewed-by: Muchun Song --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 8f6cf1221b6a..bd37a7ed32f2 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -104,6 +104,7 @@ config ARM64 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) + select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES