From patchwork Fri Jul 15 17:03:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 12919486 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6964ACCA480 for ; Fri, 15 Jul 2022 17:03:39 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.368356.599596 (Exim 4.92) (envelope-from ) id 1oCOin-0002ko-6a; Fri, 15 Jul 2022 17:03:21 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 368356.599596; Fri, 15 Jul 2022 17:03:21 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oCOin-0002kh-3h; Fri, 15 Jul 2022 17:03:21 +0000 Received: by outflank-mailman (input) for mailman id 368356; Fri, 15 Jul 2022 17:03:19 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oCOil-0002V9-8x for xen-devel@lists.xenproject.org; Fri, 15 Jul 2022 17:03:19 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oCOil-0001Ep-04; Fri, 15 Jul 2022 17:03:19 +0000 Received: from 54-240-197-232.amazon.com ([54.240.197.232] helo=dev-dsk-jgrall-1b-035652ec.eu-west-1.amazon.com) by xenbits.xenproject.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1oCOik-0007gg-OO; Fri, 15 Jul 2022 17:03:18 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-Id:Date:Subject:Cc:To:From; bh=8L8GaWN4JsHGX2p2Nk+ZEuevrv2pgtM1BqqTvQho3HU=; b=oIFyXXGVlvAjFNf6w0dtHA7myt IoggkwA7PiN1xkHSZzMzs3+FqFapQYFRpXAWCZJAcuPAZq1W5AYHrL10A8GVPu109C3ahP4jQNG+T Asse8pCVipotB0F+W2EWTyipZwHXPtf/IesJ4GKoNDHebe6sGEpqUpyNxU/H7deNrEYk=; From: Julien Grall To: xen-devel@lists.xenproject.org Cc: julien@xen.org, Julien Grall , Andrew Cooper , George Dunlap , Jan Beulich , Stefano Stabellini , Wei Liu Subject: [PATCH v2 1/3] xen: page_alloc: Don't open-code IS_ALIGNED() Date: Fri, 15 Jul 2022 18:03:10 +0100 Message-Id: <20220715170312.13931-2-julien@xen.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220715170312.13931-1-julien@xen.org> References: <20220715170312.13931-1-julien@xen.org> MIME-Version: 1.0 From: Julien Grall init_heap_pages() is using an open-code version of IS_ALIGNED(). Replace it to improve the readability of the code. No functional change intended. Signed-off-by: Julien Grall Reviewed-by: Wei Chen Acked-by: Jan Beulich --- Changes in v2: - Patch added --- xen/common/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index fe0e15429af3..078c2990041d 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -1823,7 +1823,7 @@ static void init_heap_pages( unsigned long s = mfn_x(page_to_mfn(pg + i)); unsigned long e = mfn_x(mfn_add(page_to_mfn(pg + nr_pages - 1), 1)); bool use_tail = (nid == phys_to_nid(pfn_to_paddr(e - 1))) && - !(s & ((1UL << MAX_ORDER) - 1)) && + IS_ALIGNED(s, 1UL << MAX_ORDER) && (find_first_set_bit(e) <= find_first_set_bit(s)); unsigned long n; From patchwork Fri Jul 15 17:03:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 12919487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 41DACCCA482 for ; Fri, 15 Jul 2022 17:03:39 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.368357.599606 (Exim 4.92) (envelope-from ) id 1oCOio-00030M-ED; Fri, 15 Jul 2022 17:03:22 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 368357.599606; Fri, 15 Jul 2022 17:03:22 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oCOio-00030F-Az; Fri, 15 Jul 2022 17:03:22 +0000 Received: by outflank-mailman (input) for mailman id 368357; Fri, 15 Jul 2022 17:03:20 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oCOim-0002eD-CJ for xen-devel@lists.xenproject.org; Fri, 15 Jul 2022 17:03:20 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oCOim-0001F6-6Z; Fri, 15 Jul 2022 17:03:20 +0000 Received: from 54-240-197-232.amazon.com ([54.240.197.232] helo=dev-dsk-jgrall-1b-035652ec.eu-west-1.amazon.com) by xenbits.xenproject.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1oCOil-0007gg-VD; Fri, 15 Jul 2022 17:03:20 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-Id:Date:Subject:Cc:To:From; bh=Bsjke5ID8DQGCSuhgcRoj1MKXOjC6ERn1kNwJSr/oAY=; b=2lD/AJZDGttXngLR3/Ei8/hPKh vQblHlz9cPwU18EDIXMCP+gM4miqO130gb79XxyaSwpQXu32mKjVSyUQnHM906ffE7Rc78pUdeg48 ddsIJVlVcJM+N1BORfYS3bACL/EJJR4SKqbMJAIGwLyItVT+jDAP/xRNy4DjGtmpk/CE=; From: Julien Grall To: xen-devel@lists.xenproject.org Cc: julien@xen.org, Julien Grall , Andrew Cooper , George Dunlap , Jan Beulich , Stefano Stabellini , Wei Liu Subject: [PATCH v2 2/3] xen/heap: Split init_heap_pages() in two Date: Fri, 15 Jul 2022 18:03:11 +0100 Message-Id: <20220715170312.13931-3-julien@xen.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220715170312.13931-1-julien@xen.org> References: <20220715170312.13931-1-julien@xen.org> MIME-Version: 1.0 From: Julien Grall At the moment, init_heap_pages() will call free_heap_pages() page by page. To reduce the time to initialize the heap, we will want to provide multiple pages at the same time. init_heap_pages() is now split in two parts: - init_heap_pages(): will break down the range in multiple set of contiguous pages. For now, the criteria is the pages should belong to the same NUMA node. - _init_heap_pages(): will initialize a set of pages belonging to the same NUMA node. In a follow-up patch, new requirements will be added (e.g. pages should belong to the same zone). For now the pages are still passed one by one to free_heap_pages(). Note that the comment before init_heap_pages() is heavily outdated and does not reflect the current code. So update it. This patch is a merge/rework of patches from David Woodhouse and Hongyan Xia. Signed-off-by: Julien Grall Reviewed-by: Jan Beulich --- Interestingly, I was expecting this patch to perform worse. However, from testing there is a small increase in perf. That said, I mainly plit the patch because it keeps refactoring and optimization separated. Changes in v2: - Rename init_contig_pages() to _init_heap_pages() - Fold is_contig_page() --- xen/common/page_alloc.c | 77 ++++++++++++++++++++++++++--------------- 1 file changed, 50 insertions(+), 27 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 078c2990041d..eedb2fed77c3 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -1778,16 +1778,44 @@ int query_page_offline(mfn_t mfn, uint32_t *status) } /* - * Hand the specified arbitrary page range to the specified heap zone - * checking the node_id of the previous page. If they differ and the - * latter is not on a MAX_ORDER boundary, then we reserve the page by - * not freeing it to the buddy allocator. + * This function should only be called with valid pages from the same NUMA + * node. */ +static void _init_heap_pages(const struct page_info *pg, + unsigned long nr_pages, + bool need_scrub) +{ + unsigned long s, e; + unsigned int nid = phys_to_nid(page_to_maddr(pg)); + + s = mfn_x(page_to_mfn(pg)); + e = mfn_x(mfn_add(page_to_mfn(pg + nr_pages - 1), 1)); + if ( unlikely(!avail[nid]) ) + { + bool use_tail = IS_ALIGNED(s, 1UL << MAX_ORDER) && + (find_first_set_bit(e) <= find_first_set_bit(s)); + unsigned long n; + + n = init_node_heap(nid, s, nr_pages, &use_tail); + BUG_ON(n > nr_pages); + if ( use_tail ) + e -= n; + else + s += n; + } + + while ( s < e ) + { + free_heap_pages(mfn_to_page(_mfn(s)), 0, need_scrub); + s += 1UL; + } +} + static void init_heap_pages( struct page_info *pg, unsigned long nr_pages) { unsigned long i; - bool idle_scrub = false; + bool need_scrub = scrub_debug; /* * Keep MFN 0 away from the buddy allocator to avoid crossing zone @@ -1812,35 +1840,30 @@ static void init_heap_pages( spin_unlock(&heap_lock); if ( system_state < SYS_STATE_active && opt_bootscrub == BOOTSCRUB_IDLE ) - idle_scrub = true; + need_scrub = true; - for ( i = 0; i < nr_pages; i++ ) + for ( i = 0; i < nr_pages; ) { - unsigned int nid = phys_to_nid(page_to_maddr(pg+i)); + unsigned int nid = phys_to_nid(page_to_maddr(pg)); + unsigned long left = nr_pages - i; + unsigned long contig_pages; - if ( unlikely(!avail[nid]) ) + /* + * _init_heap_pages() is only able to accept range following + * specific property (see comment on top of _init_heap_pages()). + * + * So break down the range in smaller set. + */ + for ( contig_pages = 1; contig_pages < left; contig_pages++ ) { - unsigned long s = mfn_x(page_to_mfn(pg + i)); - unsigned long e = mfn_x(mfn_add(page_to_mfn(pg + nr_pages - 1), 1)); - bool use_tail = (nid == phys_to_nid(pfn_to_paddr(e - 1))) && - IS_ALIGNED(s, 1UL << MAX_ORDER) && - (find_first_set_bit(e) <= find_first_set_bit(s)); - unsigned long n; - - n = init_node_heap(nid, mfn_x(page_to_mfn(pg + i)), nr_pages - i, - &use_tail); - BUG_ON(i + n > nr_pages); - if ( n && !use_tail ) - { - i += n - 1; - continue; - } - if ( i + n == nr_pages ) + if ( nid != (phys_to_nid(page_to_maddr(pg))) ) break; - nr_pages -= n; } - free_heap_pages(pg + i, 0, scrub_debug || idle_scrub); + _init_heap_pages(pg, contig_pages, need_scrub); + + pg += contig_pages; + i += contig_pages; } } From patchwork Fri Jul 15 17:03:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Julien Grall X-Patchwork-Id: 12919485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 41C8AC433EF for ; Fri, 15 Jul 2022 17:03:39 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.368358.599611 (Exim 4.92) (envelope-from ) id 1oCOio-00032D-MG; Fri, 15 Jul 2022 17:03:22 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 368358.599611; Fri, 15 Jul 2022 17:03:22 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oCOio-000325-I6; Fri, 15 Jul 2022 17:03:22 +0000 Received: by outflank-mailman (input) for mailman id 368358; Fri, 15 Jul 2022 17:03:21 +0000 Received: from mail.xenproject.org ([104.130.215.37]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oCOin-0002vl-Om for xen-devel@lists.xenproject.org; Fri, 15 Jul 2022 17:03:21 +0000 Received: from xenbits.xenproject.org ([104.239.192.120]) by mail.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1oCOin-0001FL-HX; Fri, 15 Jul 2022 17:03:21 +0000 Received: from 54-240-197-232.amazon.com ([54.240.197.232] helo=dev-dsk-jgrall-1b-035652ec.eu-west-1.amazon.com) by xenbits.xenproject.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1oCOin-0007gg-93; Fri, 15 Jul 2022 17:03:21 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=xen.org; s=20200302mail; h=Content-Transfer-Encoding:MIME-Version:References: In-Reply-To:Message-Id:Date:Subject:Cc:To:From; bh=pum9q+DodIOJh9CL2EYHuaYglNoP6XP3Sj591zoPRio=; b=2fBkNdi6/m3/X6Wou0bgX6JPce qLAUC8Sr8RnnatsnR64C2ES95wTF2jFTxUlGPMeFxZcxBNcm0sT2R59e5MjOfsPr8sxtua8nXdTis ltNNkjpuu9pJiVbe0hiKfeMd1C69u1+gJwhvi52nwpi1bNBquslFm07rlbLPsz73lPeo=; From: Julien Grall To: xen-devel@lists.xenproject.org Cc: julien@xen.org, Hongyan Xia , Andrew Cooper , George Dunlap , Jan Beulich , Stefano Stabellini , Wei Liu , Julien Grall Subject: [PATCH v2 3/3] xen/heap: pass order to free_heap_pages() in heap init Date: Fri, 15 Jul 2022 18:03:12 +0100 Message-Id: <20220715170312.13931-4-julien@xen.org> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220715170312.13931-1-julien@xen.org> References: <20220715170312.13931-1-julien@xen.org> MIME-Version: 1.0 From: Hongyan Xia The idea is to split the range into multiple aligned power-of-2 regions which only needs to call free_heap_pages() once each. We check the least significant set bit of the start address and use its bit index as the order of this increment. This makes sure that each increment is both power-of-2 and properly aligned, which can be safely passed to free_heap_pages(). Of course, the order also needs to be sanity checked against the upper bound and MAX_ORDER. Tested on a nested environment on c5.metal with various amount of RAM and CONFIG_DEBUG=n. Time for end_boot_allocator() to complete: Before After - 90GB: 1445 ms 96 ms - 8GB: 126 ms 8 ms - 4GB: 62 ms 4 ms Signed-off-by: Hongyan Xia Signed-off-by: Julien Grall Reviewed-by: Wei Chen Reviewed-by: Jan Beulich --- Changes in v2: - Update comment - Update the numbers. They are slightly better as is_contig_page() has been folded in init_heap_pages(). --- xen/common/page_alloc.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index eedb2fed77c3..2b99801d2ea3 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -1779,7 +1779,7 @@ int query_page_offline(mfn_t mfn, uint32_t *status) /* * This function should only be called with valid pages from the same NUMA - * node. + * node and zone. */ static void _init_heap_pages(const struct page_info *pg, unsigned long nr_pages, @@ -1806,8 +1806,22 @@ static void _init_heap_pages(const struct page_info *pg, while ( s < e ) { - free_heap_pages(mfn_to_page(_mfn(s)), 0, need_scrub); - s += 1UL; + /* + * For s == 0, we simply use the largest increment by checking the + * MSB of the region size. For s != 0, we also need to ensure that the + * chunk is properly sized to end at power-of-two alignment. We do this + * by checking the LSB of the start address and use its index as + * the increment. Both cases need to be guarded by MAX_ORDER. + * + * Note that the value of ffsl() and flsl() starts from 1 so we need + * to decrement it by 1. + */ + int inc_order = min(MAX_ORDER, flsl(e - s) - 1); + + if ( s ) + inc_order = min(inc_order, ffsl(s) - 1); + free_heap_pages(mfn_to_page(_mfn(s)), inc_order, need_scrub); + s += (1UL << inc_order); } } @@ -1844,6 +1858,9 @@ static void init_heap_pages( for ( i = 0; i < nr_pages; ) { +#ifdef CONFIG_SEPARATE_XENHEAP + unsigned int zone = page_to_zone(pg); +#endif unsigned int nid = phys_to_nid(page_to_maddr(pg)); unsigned long left = nr_pages - i; unsigned long contig_pages; @@ -1856,6 +1873,18 @@ static void init_heap_pages( */ for ( contig_pages = 1; contig_pages < left; contig_pages++ ) { + /* + * No need to check for the zone when !CONFIG_SEPARATE_XENHEAP + * because free_heap_pages() can only take power-of-two ranges + * which never cross zone boundaries. But for separate xenheap + * which is manually defined, it is possible for power-of-two + * range to cross zones. + */ +#ifdef CONFIG_SEPARATE_XENHEAP + if ( zone != page_to_zone(pg) ) + break; +#endif + if ( nid != (phys_to_nid(page_to_maddr(pg))) ) break; }