From patchwork Mon Sep 5 07:26:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Henry Wang X-Patchwork-Id: 12965713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C60AAECAAD5 for ; Mon, 5 Sep 2022 07:27:16 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.398529.639398 (Exim 4.92) (envelope-from ) id 1oV6Ve-0001pA-7Z; Mon, 05 Sep 2022 07:27:06 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 398529.639398; Mon, 05 Sep 2022 07:27:06 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oV6Ve-0001ow-4M; Mon, 05 Sep 2022 07:27:06 +0000 Received: by outflank-mailman (input) for mailman id 398529; Mon, 05 Sep 2022 07:27:05 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oV6Vd-0001oQ-6q for xen-devel@lists.xenproject.org; Mon, 05 Sep 2022 07:27:05 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id 2348637a-2cec-11ed-a016-b9edf5238543; Mon, 05 Sep 2022 09:27:03 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 43DA0ED1; Mon, 5 Sep 2022 00:27:09 -0700 (PDT) Received: from entos-skylake.shanghai.arm.com (entos-skylake.shanghai.arm.com [10.169.212.207]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id C40923F73D; Mon, 5 Sep 2022 00:27:28 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 2348637a-2cec-11ed-a016-b9edf5238543 From: Henry Wang To: xen-devel@lists.xenproject.org Cc: Henry Wang , Stefano Stabellini , Julien Grall , Bertrand Marquis , Wei Chen , Volodymyr Babchuk , Penny Zheng Subject: [PATCH v2 1/4] docs, xen/arm: Introduce reserved heap memory Date: Mon, 5 Sep 2022 07:26:32 +0000 Message-Id: <20220905072635.16294-2-Henry.Wang@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220905072635.16294-1-Henry.Wang@arm.com> References: <20220905072635.16294-1-Henry.Wang@arm.com> This commit introduces the reserved heap memory, which is parts of RAM reserved in the beginning of the boot time for heap. Firstly, since a new type of memory bank is needed for marking the memory bank solely as the heap, this commit defines `enum membank_type` and use this enum in function device_tree_get_meminfo(). Changes of code are done accordingly following the introduction of this enum. Also, this commit introduces the logic to parse the reserved heap configuration in device tree by reusing the device tree entry definition of the static memory allocation feature. If the memory bank is reserved as heap through `xen,static-heap` property in device tree `chosen` node, the memory will be marked as heap type. A documentation section is added, describing the definition of reserved heap memory and the method of enabling the reserved heap memory through device tree at boot time. Signed-off-by: Henry Wang Signed-off-by: Penny Zheng --- Changes from v1 to v2: - Rename the device tree property to xen,static-heap to avoid confusion. - Change of commit msg and doc wording, correct typo in commit msg. - Do not change the process_chosen_node() return type. - Add an empty line in make_memory_node() memory type check to improve readability. - Use enum membank_type to make the memory type cleaner. Changes from RFC to v1: - Rename the terminology to reserved heap. --- docs/misc/arm/device-tree/booting.txt | 45 +++++++++++++++++++++++++++ xen/arch/arm/bootfdt.c | 31 +++++++++++++++--- xen/arch/arm/domain_build.c | 8 +++-- xen/arch/arm/include/asm/setup.h | 7 ++++- xen/arch/arm/setup.c | 2 +- 5 files changed, 84 insertions(+), 9 deletions(-) diff --git a/docs/misc/arm/device-tree/booting.txt b/docs/misc/arm/device-tree/booting.txt index 98253414b8..ff7ca36715 100644 --- a/docs/misc/arm/device-tree/booting.txt +++ b/docs/misc/arm/device-tree/booting.txt @@ -378,3 +378,48 @@ device-tree: This will reserve a 512MB region starting at the host physical address 0x30000000 to be exclusively used by DomU1. + + +Reserved Heap Memory +==================== + +The reserved heap memory (also known as the statically-configured heap) refers +to parts of RAM reserved in the beginning of boot time for heap. The memory is +reserved by configuration in the device tree using physical address ranges. + +The reserved heap memory declared in the device tree defines the memory areas +that will be reserved to be used exclusively as heap. + +- For Arm32, since there are seperated heaps, the reserved heap will be used +for both domheap and xenheap. +- For Arm64, since there is a single heap, the defined reserved heap areas +shall always go to the heap allocator. + +The reserved heap memory is an optional feature and can be enabled by adding +below device tree properties in the `chosen` node. + +The dtb should have the following properties: + +- xen,static-heap + + Property under the top-level "chosen" node. It specifies the address + and size of Xen reserved heap memory. + +- #xen,static-heap-address-cells and #xen,static-heap-size-cells + + Specify the number of cells used for the address and size of the + "xen,static-heap" property under "chosen". + +Below is an example on how to specify the reserved heap in device tree: + + / { + chosen { + #xen,static-heap-address-cells = <0x2>; + #xen,static-heap-size-cells = <0x2>; + xen,static-heap = <0x0 0x30000000 0x0 0x40000000>; + ... + }; + }; + +RAM starting from the host physical address 0x30000000 of 1GB size will +be reserved as heap. diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c index ec81a45de9..5af71dc8ba 100644 --- a/xen/arch/arm/bootfdt.c +++ b/xen/arch/arm/bootfdt.c @@ -64,7 +64,7 @@ void __init device_tree_get_reg(const __be32 **cell, u32 address_cells, static int __init device_tree_get_meminfo(const void *fdt, int node, const char *prop_name, u32 address_cells, u32 size_cells, - void *data, bool xen_domain) + void *data, membank_type type) { const struct fdt_property *prop; unsigned int i, banks; @@ -95,7 +95,7 @@ static int __init device_tree_get_meminfo(const void *fdt, int node, continue; mem->bank[mem->nr_banks].start = start; mem->bank[mem->nr_banks].size = size; - mem->bank[mem->nr_banks].xen_domain = xen_domain; + mem->bank[mem->nr_banks].type = type; mem->nr_banks++; } @@ -185,7 +185,7 @@ static int __init process_memory_node(const void *fdt, int node, void *data) { return device_tree_get_meminfo(fdt, node, "reg", address_cells, size_cells, - data, false); + data, MEMBANK_MEMORY); } static int __init process_reserved_memory_node(const void *fdt, int node, @@ -301,6 +301,28 @@ static void __init process_chosen_node(const void *fdt, int node, paddr_t start, end; int len; + if ( fdt_get_property(fdt, node, "xen,static-heap", NULL) ) + { + u32 address_cells = device_tree_get_u32(fdt, node, + "#xen,static-heap-address-cells", 0); + u32 size_cells = device_tree_get_u32(fdt, node, + "#xen,static-heap-size-cells", 0); + + printk("Checking for reserved heap in /chosen\n"); + if ( address_cells < 1 || size_cells < 1 ) + { + printk("fdt: node `%s': invalid #xen,static-heap-address-cells or #xen,static-heap-size-cells\n", + name); + return; + } + + if ( device_tree_get_meminfo(fdt, node, "xen,static-heap", + address_cells, size_cells, + &bootinfo.reserved_mem, + MEMBANK_RSVD_HEAP) ) + return; + } + printk("Checking for initrd in /chosen\n"); prop = fdt_get_property(fdt, node, "linux,initrd-start", &len); @@ -358,7 +380,8 @@ static int __init process_domain_node(const void *fdt, int node, "#xen,static-mem-size-cells", 0); return device_tree_get_meminfo(fdt, node, "xen,static-mem", address_cells, - size_cells, &bootinfo.reserved_mem, true); + size_cells, &bootinfo.reserved_mem, + MEMBANK_XEN_DOMAIN); } static int __init early_scan_node(const void *fdt, diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index 3fd1186b53..1e46b95f0b 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -1038,9 +1038,11 @@ static int __init make_memory_node(const struct domain *d, if ( mem->nr_banks == 0 ) return -ENOENT; - /* find first memory range not bound to a Xen domain */ - for ( i = 0; i < mem->nr_banks && mem->bank[i].xen_domain; i++ ) + /* find first memory range not bound to a Xen domain nor heap */ + for ( i = 0; i < mem->nr_banks && + (mem->bank[i].type != MEMBANK_MEMORY); i++ ) ; + if ( i == mem->nr_banks ) return 0; @@ -1062,7 +1064,7 @@ static int __init make_memory_node(const struct domain *d, u64 start = mem->bank[i].start; u64 size = mem->bank[i].size; - if ( mem->bank[i].xen_domain ) + if ( mem->bank[i].type == MEMBANK_XEN_DOMAIN ) continue; dt_dprintk(" Bank %d: %#"PRIx64"->%#"PRIx64"\n", diff --git a/xen/arch/arm/include/asm/setup.h b/xen/arch/arm/include/asm/setup.h index 5815ccf8c5..d0cc556833 100644 --- a/xen/arch/arm/include/asm/setup.h +++ b/xen/arch/arm/include/asm/setup.h @@ -22,11 +22,16 @@ typedef enum { BOOTMOD_UNKNOWN } bootmodule_kind; +typedef enum { + MEMBANK_MEMORY, + MEMBANK_XEN_DOMAIN, /* whether the memory bank is bound to a Xen domain. */ + MEMBANK_RSVD_HEAP, /* whether the memory bank is reserved as heap. */ +} membank_type; struct membank { paddr_t start; paddr_t size; - bool xen_domain; /* whether the memory bank is bound to a Xen domain. */ + membank_type type; }; struct meminfo { diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index 6e0398f3f6..8d3f859982 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -644,7 +644,7 @@ static void __init init_staticmem_pages(void) for ( bank = 0 ; bank < bootinfo.reserved_mem.nr_banks; bank++ ) { - if ( bootinfo.reserved_mem.bank[bank].xen_domain ) + if ( bootinfo.reserved_mem.bank[bank].type == MEMBANK_XEN_DOMAIN ) { mfn_t bank_start = _mfn(PFN_UP(bootinfo.reserved_mem.bank[bank].start)); unsigned long bank_pages = PFN_DOWN(bootinfo.reserved_mem.bank[bank].size); From patchwork Mon Sep 5 07:26:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Henry Wang X-Patchwork-Id: 12965714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 44DE5ECAAA1 for ; Mon, 5 Sep 2022 07:27:22 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.398530.639409 (Exim 4.92) (envelope-from ) id 1oV6Vk-0002CJ-Fm; Mon, 05 Sep 2022 07:27:12 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 398530.639409; Mon, 05 Sep 2022 07:27:12 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oV6Vk-0002CC-CQ; Mon, 05 Sep 2022 07:27:12 +0000 Received: by outflank-mailman (input) for mailman id 398530; Mon, 05 Sep 2022 07:27:10 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oV6Vi-0001Wq-Es for xen-devel@lists.xenproject.org; Mon, 05 Sep 2022 07:27:10 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-flk1.inumbo.com (Halon) with ESMTP id 26ae17ad-2cec-11ed-af93-0125da4c0113; Mon, 05 Sep 2022 09:27:09 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 12251139F; Mon, 5 Sep 2022 00:27:15 -0700 (PDT) Received: from entos-skylake.shanghai.arm.com (entos-skylake.shanghai.arm.com [10.169.212.207]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id D1E7E3F73D; Mon, 5 Sep 2022 00:27:34 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 26ae17ad-2cec-11ed-af93-0125da4c0113 From: Henry Wang To: xen-devel@lists.xenproject.org Cc: Henry Wang , Stefano Stabellini , Julien Grall , Bertrand Marquis , Wei Chen , Volodymyr Babchuk Subject: [PATCH v2 2/4] xen/arm: bootfdt: Make process_chosen_node() return int Date: Mon, 5 Sep 2022 07:26:33 +0000 Message-Id: <20220905072635.16294-3-Henry.Wang@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220905072635.16294-1-Henry.Wang@arm.com> References: <20220905072635.16294-1-Henry.Wang@arm.com> At the boot time, it is saner to stop booting early if an error occurs when parsing the device tree chosen node, rather than seeing random behavior afterwards. Therefore, this commit changes the return type of the process_chosen_node() from void to int, and return correct errno based on the error type. Signed-off-by: Henry Wang --- Changes from v1 to v2: - New commit. --- xen/arch/arm/bootfdt.c | 31 +++++++++++++++++-------------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c index 5af71dc8ba..3796a4bd75 100644 --- a/xen/arch/arm/bootfdt.c +++ b/xen/arch/arm/bootfdt.c @@ -293,9 +293,9 @@ static void __init process_multiboot_node(const void *fdt, int node, kind, start, domU); } -static void __init process_chosen_node(const void *fdt, int node, - const char *name, - u32 address_cells, u32 size_cells) +static int __init process_chosen_node(const void *fdt, int node, + const char *name, + u32 address_cells, u32 size_cells) { const struct fdt_property *prop; paddr_t start, end; @@ -303,6 +303,7 @@ static void __init process_chosen_node(const void *fdt, int node, if ( fdt_get_property(fdt, node, "xen,static-heap", NULL) ) { + int rc; u32 address_cells = device_tree_get_u32(fdt, node, "#xen,static-heap-address-cells", 0); u32 size_cells = device_tree_get_u32(fdt, node, @@ -313,14 +314,14 @@ static void __init process_chosen_node(const void *fdt, int node, { printk("fdt: node `%s': invalid #xen,static-heap-address-cells or #xen,static-heap-size-cells\n", name); - return; + return -EINVAL; } - if ( device_tree_get_meminfo(fdt, node, "xen,static-heap", + rc = device_tree_get_meminfo(fdt, node, "xen,static-heap", address_cells, size_cells, - &bootinfo.reserved_mem, - MEMBANK_RSVD_HEAP) ) - return; + &bootinfo.reserved_mem, MEMBANK_RSVD_HEAP); + if ( rc ) + return rc; } printk("Checking for initrd in /chosen\n"); @@ -328,11 +329,11 @@ static void __init process_chosen_node(const void *fdt, int node, prop = fdt_get_property(fdt, node, "linux,initrd-start", &len); if ( !prop ) /* No initrd present. */ - return; + return 0; if ( len != sizeof(u32) && len != sizeof(u64) ) { printk("linux,initrd-start property has invalid length %d\n", len); - return; + return -EINVAL; } start = dt_read_number((void *)&prop->data, dt_size_to_cells(len)); @@ -340,12 +341,12 @@ static void __init process_chosen_node(const void *fdt, int node, if ( !prop ) { printk("linux,initrd-end not present but -start was\n"); - return; + return -EINVAL; } if ( len != sizeof(u32) && len != sizeof(u64) ) { printk("linux,initrd-end property has invalid length %d\n", len); - return; + return -EINVAL; } end = dt_read_number((void *)&prop->data, dt_size_to_cells(len)); @@ -353,12 +354,14 @@ static void __init process_chosen_node(const void *fdt, int node, { printk("linux,initrd limits invalid: %"PRIpaddr" >= %"PRIpaddr"\n", start, end); - return; + return -EINVAL; } printk("Initrd %"PRIpaddr"-%"PRIpaddr"\n", start, end); add_boot_module(BOOTMOD_RAMDISK, start, end-start, false); + + return 0; } static int __init process_domain_node(const void *fdt, int node, @@ -406,7 +409,7 @@ static int __init early_scan_node(const void *fdt, device_tree_node_compatible(fdt, node, "multiboot,module" ))) process_multiboot_node(fdt, node, name, address_cells, size_cells); else if ( depth == 1 && device_tree_node_matches(fdt, node, "chosen") ) - process_chosen_node(fdt, node, name, address_cells, size_cells); + rc = process_chosen_node(fdt, node, name, address_cells, size_cells); else if ( depth == 2 && device_tree_node_compatible(fdt, node, "xen,domain") ) rc = process_domain_node(fdt, node, name, address_cells, size_cells); From patchwork Mon Sep 5 07:26:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Henry Wang X-Patchwork-Id: 12965715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 816FCECAAA1 for ; Mon, 5 Sep 2022 07:27:25 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.398534.639419 (Exim 4.92) (envelope-from ) id 1oV6Vp-0002bj-OR; Mon, 05 Sep 2022 07:27:17 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 398534.639419; Mon, 05 Sep 2022 07:27:17 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oV6Vp-0002bZ-LE; Mon, 05 Sep 2022 07:27:17 +0000 Received: by outflank-mailman (input) for mailman id 398534; Mon, 05 Sep 2022 07:27:15 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oV6Vn-0001oQ-M0 for xen-devel@lists.xenproject.org; Mon, 05 Sep 2022 07:27:15 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id 29a410c7-2cec-11ed-a016-b9edf5238543; Mon, 05 Sep 2022 09:27:14 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EA22BED1; Mon, 5 Sep 2022 00:27:19 -0700 (PDT) Received: from entos-skylake.shanghai.arm.com (entos-skylake.shanghai.arm.com [10.169.212.207]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id B4F9A3F73D; Mon, 5 Sep 2022 00:27:39 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 29a410c7-2cec-11ed-a016-b9edf5238543 From: Henry Wang To: xen-devel@lists.xenproject.org Cc: Henry Wang , Stefano Stabellini , Julien Grall , Bertrand Marquis , Wei Chen , Volodymyr Babchuk Subject: [PATCH v2 3/4] xen/arm: Handle reserved heap pages in boot and heap allocator Date: Mon, 5 Sep 2022 07:26:34 +0000 Message-Id: <20220905072635.16294-4-Henry.Wang@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220905072635.16294-1-Henry.Wang@arm.com> References: <20220905072635.16294-1-Henry.Wang@arm.com> This commit firstly adds a bool field `reserved_heap` to bootinfo. This newly introduced field is set at the device tree parsing time if the reserved heap ranges are defined in the device tree chosen node. For Arm32, In `setup_mm`, if the reserved heap is enabled, we use the reserved heap region for both domheap and xenheap allocation. Note that the xenheap on Arm32 should be always contiguous, so also add a helper fit_xenheap_in_reserved_heap() for Arm32 to find the required xenheap in the reserved heap regions. For Arm64, In `setup_mm`, if the reserved heap is enabled and used, we make sure that only these reserved heap pages are added to the boot allocator. These reserved heap pages in the boot allocator are added to the heap allocator at `end_boot_allocator()`. If the reserved heap is disabled, we stick to current page allocation strategy at boot time. Also, take the chance to correct a "double not" print in Arm32 `setup_mm()` and replace the open-coding address ~0 by INVALID_PADDR. Signed-off-by: Henry Wang --- Changes from v1 to v2: - Move the global bool `reserved_heap` to bootinfo. - Replace the open open-coding address ~0 by INVALID_PADDR. - Do not use reverted logic in heap_pages calculation. - Remove unused Arm32 reserved_heap_start variable. - Decouple the arm32 reserved heap too small size check with region end check. - Reuse the arm32 original xenheap finding logic with the new helper to make sure xenheap on arm32 is contiguous. Changes from RFC to v1: - Rebase on top of latest `setup_mm()` changes. - Added Arm32 logic in `setup_mm()`. --- xen/arch/arm/bootfdt.c | 2 + xen/arch/arm/include/asm/setup.h | 1 + xen/arch/arm/setup.c | 116 +++++++++++++++++++++++++++---- 3 files changed, 104 insertions(+), 15 deletions(-) diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c index 3796a4bd75..616bf5ce47 100644 --- a/xen/arch/arm/bootfdt.c +++ b/xen/arch/arm/bootfdt.c @@ -322,6 +322,8 @@ static int __init process_chosen_node(const void *fdt, int node, &bootinfo.reserved_mem, MEMBANK_RSVD_HEAP); if ( rc ) return rc; + + bootinfo.reserved_heap = true; } printk("Checking for initrd in /chosen\n"); diff --git a/xen/arch/arm/include/asm/setup.h b/xen/arch/arm/include/asm/setup.h index d0cc556833..22fb950bc8 100644 --- a/xen/arch/arm/include/asm/setup.h +++ b/xen/arch/arm/include/asm/setup.h @@ -82,6 +82,7 @@ struct bootinfo { #ifdef CONFIG_ACPI struct meminfo acpi; #endif + bool reserved_heap; }; struct map_range_data diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index 8d3f859982..0b4f7cb909 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -556,6 +556,43 @@ static paddr_t __init consider_modules(paddr_t s, paddr_t e, } return e; } + +/* + * Find the contiguous xenheap region that fits in the reserved heap region with + * required size and alignment, and return the end address of xenheap. + */ +static paddr_t __init fit_xenheap_in_reserved_heap(uint32_t size, paddr_t align) +{ + int i; + paddr_t end = 0, aligned_start, aligned_end; + paddr_t bank_start, bank_size, bank_end; + + for ( i = 0 ; i < bootinfo.reserved_mem.nr_banks; i++ ) + { + if ( bootinfo.reserved_mem.bank[i].type == MEMBANK_RSVD_HEAP ) + { + bank_start = bootinfo.reserved_mem.bank[i].start; + bank_size = bootinfo.reserved_mem.bank[i].size; + bank_end = bank_start + bank_size; + + if ( bank_size < size ) + continue; + + aligned_end = bank_end & ~(align - 1); + aligned_start = (aligned_end - size) & ~(align - 1); + + if ( aligned_start > bank_start ) + /* + * Arm32 allocates xenheap from higher address to lower, so if + * there are multiple memory banks that satisfy the requirement, + * use the highest bank. + */ + end = max(end, aligned_end); + } + } + + return end; +} #endif /* @@ -713,8 +750,9 @@ static void __init populate_boot_allocator(void) #ifdef CONFIG_ARM_32 static void __init setup_mm(void) { - paddr_t ram_start, ram_end, ram_size, e; - unsigned long ram_pages; + paddr_t ram_start, ram_end, ram_size, e, bank_start, bank_end, bank_size; + paddr_t reserved_heap_end = 0, reserved_heap_size = 0; + unsigned long ram_pages, reserved_heap_pages = 0; unsigned long heap_pages, xenheap_pages, domheap_pages; unsigned int i; const uint32_t ctr = READ_CP32(CTR); @@ -734,9 +772,9 @@ static void __init setup_mm(void) for ( i = 1; i < bootinfo.mem.nr_banks; i++ ) { - paddr_t bank_start = bootinfo.mem.bank[i].start; - paddr_t bank_size = bootinfo.mem.bank[i].size; - paddr_t bank_end = bank_start + bank_size; + bank_start = bootinfo.mem.bank[i].start; + bank_size = bootinfo.mem.bank[i].size; + bank_end = bank_start + bank_size; ram_size = ram_size + bank_size; ram_start = min(ram_start,bank_start); @@ -745,19 +783,42 @@ static void __init setup_mm(void) total_pages = ram_pages = ram_size >> PAGE_SHIFT; + if ( bootinfo.reserved_heap ) + { + for ( i = 0 ; i < bootinfo.reserved_mem.nr_banks; i++ ) + { + if ( bootinfo.reserved_mem.bank[i].type == MEMBANK_RSVD_HEAP ) + { + bank_start = bootinfo.reserved_mem.bank[i].start; + bank_size = bootinfo.reserved_mem.bank[i].size; + bank_end = bank_start + bank_size; + + reserved_heap_size += bank_size; + reserved_heap_end = max(reserved_heap_end, bank_end); + } + } + + reserved_heap_pages = reserved_heap_size >> PAGE_SHIFT; + if ( reserved_heap_pages < 32<<(20-PAGE_SHIFT) ) + panic("Too small reserved heap region, should be at least 32M\n"); + } + /* * If the user has not requested otherwise via the command line * then locate the xenheap using these constraints: * * - must be 32 MiB aligned * - must not include Xen itself or the boot modules - * - must be at most 1GB or 1/32 the total RAM in the system if less + * - must be at most 1GB or 1/32 the total RAM in the system + * (there is no reserved heap) or 1/32 the total reserved + * heap region (there is reserved heap) if less * - must be at least 32M * * We try to allocate the largest xenheap possible within these * constraints. */ - heap_pages = ram_pages; + heap_pages = bootinfo.reserved_heap ? reserved_heap_pages : ram_pages; + if ( opt_xenheap_megabytes ) xenheap_pages = opt_xenheap_megabytes << (20-PAGE_SHIFT); else @@ -767,9 +828,15 @@ static void __init setup_mm(void) xenheap_pages = min(xenheap_pages, 1UL<<(30-PAGE_SHIFT)); } + /* + * On Arm32, xenheap must be contiguous, look for one of the region + * that matches the above-mentioned xenheap constraints. + */ do { - e = consider_modules(ram_start, ram_end, + e = bootinfo.reserved_heap ? + fit_xenheap_in_reserved_heap(pfn_to_paddr(xenheap_pages), 32<<20) : + consider_modules(ram_start, ram_end, pfn_to_paddr(xenheap_pages), 32<<20, 0); if ( e ) @@ -779,7 +846,7 @@ static void __init setup_mm(void) } while ( !opt_xenheap_megabytes && xenheap_pages > 32<<(20-PAGE_SHIFT) ); if ( ! e ) - panic("Not not enough space for xenheap\n"); + panic("Not enough space for xenheap\n"); domheap_pages = heap_pages - xenheap_pages; @@ -824,9 +891,9 @@ static void __init setup_mm(void) static void __init setup_mm(void) { const struct meminfo *banks = &bootinfo.mem; - paddr_t ram_start = ~0; - paddr_t ram_end = 0; - paddr_t ram_size = 0; + paddr_t ram_start = INVALID_PADDR, bank_start = INVALID_PADDR; + paddr_t ram_end = 0, bank_end = 0; + paddr_t ram_size = 0, bank_size = 0; unsigned int i; init_pdx(); @@ -835,17 +902,36 @@ static void __init setup_mm(void) * We need some memory to allocate the page-tables used for the xenheap * mappings. But some regions may contain memory already allocated * for other uses (e.g. modules, reserved-memory...). - * + * If there are non-empty reserved heap regions, (only) add these regions + * in the boot allocator. + */ + if ( bootinfo.reserved_heap ) + { + for ( i = 0 ; i < bootinfo.reserved_mem.nr_banks; i++ ) + { + if ( bootinfo.reserved_mem.bank[i].type == MEMBANK_RSVD_HEAP ) + { + bank_start = bootinfo.reserved_mem.bank[i].start; + bank_size = bootinfo.reserved_mem.bank[i].size; + bank_end = bank_start + bank_size; + + init_boot_pages(bank_start, bank_end); + } + } + } + /* + * No reserved heap regions: * For simplicity, add all the free regions in the boot allocator. */ - populate_boot_allocator(); + else + populate_boot_allocator(); total_pages = 0; for ( i = 0; i < banks->nr_banks; i++ ) { const struct membank *bank = &banks->bank[i]; - paddr_t bank_end = bank->start + bank->size; + bank_end = bank->start + bank->size; ram_size = ram_size + bank->size; ram_start = min(ram_start, bank->start); From patchwork Mon Sep 5 07:26:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Henry Wang X-Patchwork-Id: 12965716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79628ECAAD5 for ; Mon, 5 Sep 2022 07:27:29 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.398536.639431 (Exim 4.92) (envelope-from ) id 1oV6Vt-00030G-7k; Mon, 05 Sep 2022 07:27:21 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 398536.639431; Mon, 05 Sep 2022 07:27:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oV6Vt-000305-40; Mon, 05 Sep 2022 07:27:21 +0000 Received: by outflank-mailman (input) for mailman id 398536; Mon, 05 Sep 2022 07:27:19 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oV6Vr-0001oQ-PV for xen-devel@lists.xenproject.org; Mon, 05 Sep 2022 07:27:19 +0000 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by se1-gles-sth1.inumbo.com (Halon) with ESMTP id 2c22bfe2-2cec-11ed-a016-b9edf5238543; Mon, 05 Sep 2022 09:27:18 +0200 (CEST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 23FEDED1; Mon, 5 Sep 2022 00:27:24 -0700 (PDT) Received: from entos-skylake.shanghai.arm.com (entos-skylake.shanghai.arm.com [10.169.212.207]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id E3FBF3F73D; Mon, 5 Sep 2022 00:27:43 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 2c22bfe2-2cec-11ed-a016-b9edf5238543 From: Henry Wang To: xen-devel@lists.xenproject.org Cc: Henry Wang , Stefano Stabellini , Julien Grall , Bertrand Marquis , Wei Chen , Volodymyr Babchuk Subject: [PATCH v2 4/4] xen/arm: mm: Rename xenheap_* variable to directmap_* Date: Mon, 5 Sep 2022 07:26:35 +0000 Message-Id: <20220905072635.16294-5-Henry.Wang@arm.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20220905072635.16294-1-Henry.Wang@arm.com> References: <20220905072635.16294-1-Henry.Wang@arm.com> With the reserved heap setup, keep using xenheap_* in the function setup_xenheap_mappings() will make the code confusing to read, because we always need to map the full RAM on Arm64. Therefore, renaming all "xenheap_*" variables to "directmap_*" to make clear the area is used to access the RAM easily. On Arm32, only the xenheap is direct mapped today. So the renaming to "directmap_*" would be still valid for Arm32. No functional change is intended. Signed-off-by: Henry Wang --- Changes from v1 to v2: - New commit. --- xen/arch/arm/include/asm/config.h | 2 +- xen/arch/arm/include/asm/mm.h | 22 +++++++++++----------- xen/arch/arm/mm.c | 24 ++++++++++++------------ xen/arch/arm/setup.c | 27 ++++++++++++++------------- 4 files changed, 38 insertions(+), 37 deletions(-) diff --git a/xen/arch/arm/include/asm/config.h b/xen/arch/arm/include/asm/config.h index 2fafb9f228..0fefed1b8a 100644 --- a/xen/arch/arm/include/asm/config.h +++ b/xen/arch/arm/include/asm/config.h @@ -160,7 +160,7 @@ #define DIRECTMAP_SIZE (SLOT0_ENTRY_SIZE * (265-256)) #define DIRECTMAP_VIRT_END (DIRECTMAP_VIRT_START + DIRECTMAP_SIZE - 1) -#define XENHEAP_VIRT_START xenheap_virt_start +#define XENHEAP_VIRT_START directmap_virt_start #define HYPERVISOR_VIRT_END DIRECTMAP_VIRT_END diff --git a/xen/arch/arm/include/asm/mm.h b/xen/arch/arm/include/asm/mm.h index da25251cda..1dcb0a093a 100644 --- a/xen/arch/arm/include/asm/mm.h +++ b/xen/arch/arm/include/asm/mm.h @@ -150,19 +150,19 @@ struct page_info #define _PGC_need_scrub _PGC_allocated #define PGC_need_scrub PGC_allocated -extern mfn_t xenheap_mfn_start, xenheap_mfn_end; -extern vaddr_t xenheap_virt_end; +extern mfn_t directmap_mfn_start, directmap_mfn_end; +extern vaddr_t directmap_virt_end; #ifdef CONFIG_ARM_64 -extern vaddr_t xenheap_virt_start; -extern unsigned long xenheap_base_pdx; +extern vaddr_t directmap_virt_start; +extern unsigned long directmap_base_pdx; #endif #ifdef CONFIG_ARM_32 #define is_xen_heap_page(page) is_xen_heap_mfn(page_to_mfn(page)) #define is_xen_heap_mfn(mfn) ({ \ unsigned long mfn_ = mfn_x(mfn); \ - (mfn_ >= mfn_x(xenheap_mfn_start) && \ - mfn_ < mfn_x(xenheap_mfn_end)); \ + (mfn_ >= mfn_x(directmap_mfn_start) && \ + mfn_ < mfn_x(directmap_mfn_end)); \ }) #else #define is_xen_heap_page(page) ((page)->count_info & PGC_xen_heap) @@ -263,16 +263,16 @@ static inline paddr_t __virt_to_maddr(vaddr_t va) static inline void *maddr_to_virt(paddr_t ma) { ASSERT(is_xen_heap_mfn(maddr_to_mfn(ma))); - ma -= mfn_to_maddr(xenheap_mfn_start); + ma -= mfn_to_maddr(directmap_mfn_start); return (void *)(unsigned long) ma + XENHEAP_VIRT_START; } #else static inline void *maddr_to_virt(paddr_t ma) { - ASSERT((mfn_to_pdx(maddr_to_mfn(ma)) - xenheap_base_pdx) < + ASSERT((mfn_to_pdx(maddr_to_mfn(ma)) - directmap_base_pdx) < (DIRECTMAP_SIZE >> PAGE_SHIFT)); return (void *)(XENHEAP_VIRT_START - - (xenheap_base_pdx << PAGE_SHIFT) + + (directmap_base_pdx << PAGE_SHIFT) + ((ma & ma_va_bottom_mask) | ((ma & ma_top_mask) >> pfn_pdx_hole_shift))); } @@ -315,10 +315,10 @@ static inline struct page_info *virt_to_page(const void *v) unsigned long pdx; ASSERT(va >= XENHEAP_VIRT_START); - ASSERT(va < xenheap_virt_end); + ASSERT(va < directmap_virt_end); pdx = (va - XENHEAP_VIRT_START) >> PAGE_SHIFT; - pdx += mfn_to_pdx(xenheap_mfn_start); + pdx += mfn_to_pdx(directmap_mfn_start); return frame_table + pdx - frametable_base_pdx; } diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index c81c706c8b..7d6814caf2 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -132,12 +132,12 @@ uint64_t init_ttbr; static paddr_t phys_offset; /* Limits of the Xen heap */ -mfn_t xenheap_mfn_start __read_mostly = INVALID_MFN_INITIALIZER; -mfn_t xenheap_mfn_end __read_mostly; -vaddr_t xenheap_virt_end __read_mostly; +mfn_t directmap_mfn_start __read_mostly = INVALID_MFN_INITIALIZER; +mfn_t directmap_mfn_end __read_mostly; +vaddr_t directmap_virt_end __read_mostly; #ifdef CONFIG_ARM_64 -vaddr_t xenheap_virt_start __read_mostly; -unsigned long xenheap_base_pdx __read_mostly; +vaddr_t directmap_virt_start __read_mostly; +unsigned long directmap_base_pdx __read_mostly; #endif unsigned long frametable_base_pdx __read_mostly; @@ -609,7 +609,7 @@ void __init setup_xenheap_mappings(unsigned long base_mfn, panic("Unable to setup the xenheap mappings.\n"); /* Record where the xenheap is, for translation routines. */ - xenheap_virt_end = XENHEAP_VIRT_START + nr_mfns * PAGE_SIZE; + directmap_virt_end = XENHEAP_VIRT_START + nr_mfns * PAGE_SIZE; } #else /* CONFIG_ARM_64 */ void __init setup_xenheap_mappings(unsigned long base_mfn, @@ -618,12 +618,12 @@ void __init setup_xenheap_mappings(unsigned long base_mfn, int rc; /* First call sets the xenheap physical and virtual offset. */ - if ( mfn_eq(xenheap_mfn_start, INVALID_MFN) ) + if ( mfn_eq(directmap_mfn_start, INVALID_MFN) ) { unsigned long mfn_gb = base_mfn & ~((FIRST_SIZE >> PAGE_SHIFT) - 1); - xenheap_mfn_start = _mfn(base_mfn); - xenheap_base_pdx = mfn_to_pdx(_mfn(base_mfn)); + directmap_mfn_start = _mfn(base_mfn); + directmap_base_pdx = mfn_to_pdx(_mfn(base_mfn)); /* * The base address may not be aligned to the first level * size (e.g. 1GB when using 4KB pages). This would prevent @@ -633,13 +633,13 @@ void __init setup_xenheap_mappings(unsigned long base_mfn, * Prevent that by offsetting the start of the xenheap virtual * address. */ - xenheap_virt_start = DIRECTMAP_VIRT_START + + directmap_virt_start = DIRECTMAP_VIRT_START + (base_mfn - mfn_gb) * PAGE_SIZE; } - if ( base_mfn < mfn_x(xenheap_mfn_start) ) + if ( base_mfn < mfn_x(directmap_mfn_start) ) panic("cannot add xenheap mapping at %lx below heap start %lx\n", - base_mfn, mfn_x(xenheap_mfn_start)); + base_mfn, mfn_x(directmap_mfn_start)); rc = map_pages_to_xen((vaddr_t)__mfn_to_virt(base_mfn), _mfn(base_mfn), nr_mfns, diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c index 0b4f7cb909..399a695ff6 100644 --- a/xen/arch/arm/setup.c +++ b/xen/arch/arm/setup.c @@ -733,11 +733,11 @@ static void __init populate_boot_allocator(void) #ifdef CONFIG_ARM_32 /* Avoid the xenheap */ - if ( s < mfn_to_maddr(xenheap_mfn_end) && - mfn_to_maddr(xenheap_mfn_start) < e ) + if ( s < mfn_to_maddr(directmap_mfn_end) && + mfn_to_maddr(directmap_mfn_start) < e ) { - e = mfn_to_maddr(xenheap_mfn_start); - n = mfn_to_maddr(xenheap_mfn_end); + e = mfn_to_maddr(directmap_mfn_start); + n = mfn_to_maddr(directmap_mfn_end); } #endif @@ -859,15 +859,16 @@ static void __init setup_mm(void) * We need some memory to allocate the page-tables used for the * xenheap mappings. So populate the boot allocator first. * - * This requires us to set xenheap_mfn_{start, end} first so the Xenheap + * Note that currently xenheap is direct mapped on Arm32. + * This requires us to set directmap_mfn_{start, end} first so the Xenheap * region can be avoided. */ - xenheap_mfn_start = _mfn((e >> PAGE_SHIFT) - xenheap_pages); - xenheap_mfn_end = mfn_add(xenheap_mfn_start, xenheap_pages); + directmap_mfn_start = _mfn((e >> PAGE_SHIFT) - xenheap_pages); + directmap_mfn_end = mfn_add(directmap_mfn_start, xenheap_pages); populate_boot_allocator(); - setup_xenheap_mappings(mfn_x(xenheap_mfn_start), xenheap_pages); + setup_xenheap_mappings(mfn_x(directmap_mfn_start), xenheap_pages); /* Frame table covers all of RAM region, including holes */ setup_frametable_mappings(ram_start, ram_end); @@ -882,8 +883,8 @@ static void __init setup_mm(void) smp_processor_id()); /* Add xenheap memory that was not already added to the boot allocator. */ - init_xenheap_pages(mfn_to_maddr(xenheap_mfn_start), - mfn_to_maddr(xenheap_mfn_end)); + init_xenheap_pages(mfn_to_maddr(directmap_mfn_start), + mfn_to_maddr(directmap_mfn_end)); init_staticmem_pages(); } @@ -943,9 +944,9 @@ static void __init setup_mm(void) total_pages += ram_size >> PAGE_SHIFT; - xenheap_virt_end = XENHEAP_VIRT_START + ram_end - ram_start; - xenheap_mfn_start = maddr_to_mfn(ram_start); - xenheap_mfn_end = maddr_to_mfn(ram_end); + directmap_virt_end = XENHEAP_VIRT_START + ram_end - ram_start; + directmap_mfn_start = maddr_to_mfn(ram_start); + directmap_mfn_end = maddr_to_mfn(ram_end); setup_frametable_mappings(ram_start, ram_end); max_page = PFN_DOWN(ram_end);