From patchwork Thu Jun 22 18:57:04 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Ostrovsky X-Patchwork-Id: 9805111 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 7544D60329 for ; Thu, 22 Jun 2017 18:58:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 64FE6283DA for ; Thu, 22 Jun 2017 18:58:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 599FD286E1; Thu, 22 Jun 2017 18:58:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8D26F283DA for ; Thu, 22 Jun 2017 18:58:09 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dO7GY-0005Pf-9L; Thu, 22 Jun 2017 18:55:42 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dO7GW-0005Om-Ky for xen-devel@lists.xen.org; Thu, 22 Jun 2017 18:55:40 +0000 Received: from [193.109.254.147] by server-3.bemta-6.messagelabs.com id 77/F7-03058-CA21C495; Thu, 22 Jun 2017 18:55:40 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrHLMWRWlGSWpSXmKPExsUyZ7p8oO5qIZ9 Ig30zlC2WfFzM4sDocXT3b6YAxijWzLyk/IoE1oxp03rYC6bYV/R3d7M2MB7Q7WLk4hASmMgk 0fv5FguE85tR4m3HX2YIZyOjxLyfi5ggnB5GiTXTDrF1MXJysAkYSZw9Op0RxBYRkJa49vkyI 0gRs0ADk8TzcwfBEsIC3hJPbjawgNgsAqoSi/btBbN5Bbwk/vZuARskIaAgMeXhe2YQmxOo/v Gpt2C9QkA1S9+uZYeoMZT4vHEp8wRGvgWMDKsYNYpTi8pSi3SNLPSSijLTM0pyEzNzdA0NzPR yU4uLE9NTcxKTivWS83M3MQLDhQEIdjCeXxt4iFGSg0lJlHdzsXekEF9SfkplRmJxRnxRaU5q 8SFGGQ4OJQne1YI+kUKCRanpqRVpmTnAwIVJS3DwKInwfuEDSvMWFyTmFmemQ6ROMSpKifPuA +kTAElklObBtcGi5RKjrJQwLyPQIUI8BalFuZklqPKvGMU5GJWEeb+DTOHJzCuBm/4KaDET0O IXRzxAFpckIqSkGhjnKi29ktX90Do47nqq9v6/cU9OaE7b+TG9SWVKn0jm7M8SU6tqeovfPVv wIuTn4oQjYtV+D+WcGb3u72ZM5bcPt7yU43PVtKuM+fiHzDmMqguY5BYYN4hvD3x7ejFfoenT /Koe9h/xHXeWyHoIllrrzVcKi2HsMC/e+Xl1DEOhS/rMzxs+vVViKc5INNRiLipOBADRxp+Wk QIAAA== X-Env-Sender: boris.ostrovsky@oracle.com X-Msg-Ref: server-16.tower-27.messagelabs.com!1498157737!108707587!1 X-Originating-IP: [156.151.31.81] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTU2LjE1MS4zMS44MSA9PiAyODgzMzk=\n X-StarScan-Received: X-StarScan-Version: 9.4.19; banners=-,-,- X-VirusChecked: Checked Received: (qmail 57280 invoked from network); 22 Jun 2017 18:55:39 -0000 Received: from userp1040.oracle.com (HELO userp1040.oracle.com) (156.151.31.81) by server-16.tower-27.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 22 Jun 2017 18:55:39 -0000 Received: from userv0022.oracle.com (userv0022.oracle.com [156.151.31.74]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v5MItVHQ011631 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Jun 2017 18:55:31 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userv0022.oracle.com (8.14.4/8.14.4) with ESMTP id v5MItVtD017379 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 22 Jun 2017 18:55:31 GMT Received: from abhmp0015.oracle.com (abhmp0015.oracle.com [141.146.116.21]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id v5MItUD5005579; Thu, 22 Jun 2017 18:55:30 GMT Received: from ovs104.us.oracle.com (/10.149.76.204) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 22 Jun 2017 11:55:30 -0700 From: Boris Ostrovsky To: xen-devel@lists.xen.org Date: Thu, 22 Jun 2017 14:57:04 -0400 Message-Id: <1498157830-21845-3-git-send-email-boris.ostrovsky@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1498157830-21845-1-git-send-email-boris.ostrovsky@oracle.com> References: <1498157830-21845-1-git-send-email-boris.ostrovsky@oracle.com> X-Source-IP: userv0022.oracle.com [156.151.31.74] Cc: sstabellini@kernel.org, wei.liu2@citrix.com, George.Dunlap@eu.citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, tim@xen.org, jbeulich@suse.com, Boris Ostrovsky Subject: [Xen-devel] [PATCH v5 2/8] mm: Extract allocation loop from alloc_heap_pages() X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This will make code a bit more readable, especially with changes that will be introduced in subsequent patches. Signed-off-by: Boris Ostrovsky Acked-by: Jan Beulich --- Changes in v5: * Constified get_free_buddy()'s struct domain argument * Dropped request local variable in get_free_buddy(). Because of rebasing there were few more changes in this patch so I decided not to keep Jan's ACK. xen/common/page_alloc.c | 143 ++++++++++++++++++++++++++---------------------- 1 file changed, 79 insertions(+), 64 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 570d1f7..89fe3ce 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -694,22 +694,15 @@ static void page_list_add_scrub(struct page_info *pg, unsigned int node, page_list_add(pg, &heap(node, zone, order)); } -/* Allocate 2^@order contiguous pages. */ -static struct page_info *alloc_heap_pages( - unsigned int zone_lo, unsigned int zone_hi, - unsigned int order, unsigned int memflags, - struct domain *d) +static struct page_info *get_free_buddy(unsigned int zone_lo, + unsigned int zone_hi, + unsigned int order, unsigned int memflags, + const struct domain *d) { - unsigned int i, j, zone = 0, nodemask_retry = 0; nodeid_t first_node, node = MEMF_get_node(memflags), req_node = node; - unsigned long request = 1UL << order; - struct page_info *pg, *first_dirty_pg = NULL; - nodemask_t nodemask = (d != NULL ) ? d->node_affinity : node_online_map; - bool_t need_tlbflush = 0; - uint32_t tlbflush_timestamp = 0; - - /* Make sure there are enough bits in memflags for nodeID. */ - BUILD_BUG_ON((_MEMF_bits - _MEMF_node) < (8 * sizeof(nodeid_t))); + nodemask_t nodemask = d ? d->node_affinity : node_online_map; + unsigned int j, zone, nodemask_retry = 0; + struct page_info *pg; if ( node == NUMA_NO_NODE ) { @@ -725,34 +718,6 @@ static struct page_info *alloc_heap_pages( first_node = node; ASSERT(node < MAX_NUMNODES); - ASSERT(zone_lo <= zone_hi); - ASSERT(zone_hi < NR_ZONES); - - if ( unlikely(order > MAX_ORDER) ) - return NULL; - - spin_lock(&heap_lock); - - /* - * Claimed memory is considered unavailable unless the request - * is made by a domain with sufficient unclaimed pages. - */ - if ( (outstanding_claims + request > - total_avail_pages + tmem_freeable_pages()) && - ((memflags & MEMF_no_refcount) || - !d || d->outstanding_pages < request) ) - goto not_found; - - /* - * TMEM: When available memory is scarce due to tmem absorbing it, allow - * only mid-size allocations to avoid worst of fragmentation issues. - * Others try tmem pools then fail. This is a workaround until all - * post-dom0-creation-multi-page allocations can be eliminated. - */ - if ( ((order == 0) || (order >= 9)) && - (total_avail_pages <= midsize_alloc_zone_pages) && - tmem_freeable_pages() ) - goto try_tmem; /* * Start with requested node, but exhaust all node memory in requested @@ -764,17 +729,17 @@ static struct page_info *alloc_heap_pages( zone = zone_hi; do { /* Check if target node can support the allocation. */ - if ( !avail[node] || (avail[node][zone] < request) ) + if ( !avail[node] || (avail[node][zone] < (1UL << order)) ) continue; /* Find smallest order which can satisfy the request. */ for ( j = order; j <= MAX_ORDER; j++ ) if ( (pg = page_list_remove_head(&heap(node, zone, j))) ) - goto found; + return pg; } while ( zone-- > zone_lo ); /* careful: unsigned zone may wrap */ if ( (memflags & MEMF_exact_node) && req_node != NUMA_NO_NODE ) - goto not_found; + return NULL; /* Pick next node. */ if ( !node_isset(node, nodemask) ) @@ -791,39 +756,89 @@ static struct page_info *alloc_heap_pages( { /* When we have tried all in nodemask, we fall back to others. */ if ( (memflags & MEMF_exact_node) || nodemask_retry++ ) - goto not_found; + return NULL; nodes_andnot(nodemask, node_online_map, nodemask); first_node = node = first_node(nodemask); if ( node >= MAX_NUMNODES ) - goto not_found; + return NULL; } } +} - try_tmem: - /* Try to free memory from tmem */ - if ( (pg = tmem_relinquish_pages(order, memflags)) != NULL ) +/* Allocate 2^@order contiguous pages. */ +static struct page_info *alloc_heap_pages( + unsigned int zone_lo, unsigned int zone_hi, + unsigned int order, unsigned int memflags, + struct domain *d) +{ + nodeid_t node; + unsigned int i, buddy_order, zone; + unsigned long request = 1UL << order; + struct page_info *pg, *first_dirty_pg = NULL; + bool_t need_tlbflush = 0; + uint32_t tlbflush_timestamp = 0; + + /* Make sure there are enough bits in memflags for nodeID. */ + BUILD_BUG_ON((_MEMF_bits - _MEMF_node) < (8 * sizeof(nodeid_t))); + + ASSERT(zone_lo <= zone_hi); + ASSERT(zone_hi < NR_ZONES); + + if ( unlikely(order > MAX_ORDER) ) + return NULL; + + spin_lock(&heap_lock); + + /* + * Claimed memory is considered unavailable unless the request + * is made by a domain with sufficient unclaimed pages. + */ + if ( (outstanding_claims + request > + total_avail_pages + tmem_freeable_pages()) && + ((memflags & MEMF_no_refcount) || + !d || d->outstanding_pages < request) ) { - /* reassigning an already allocated anonymous heap page */ spin_unlock(&heap_lock); - return pg; + return NULL; + } + + /* + * TMEM: When available memory is scarce due to tmem absorbing it, allow + * only mid-size allocations to avoid worst of fragmentation issues. + * Others try tmem pools then fail. This is a workaround until all + * post-dom0-creation-multi-page allocations can be eliminated. + */ + if ( ((order == 0) || (order >= 9)) && + (total_avail_pages <= midsize_alloc_zone_pages) && + tmem_freeable_pages() ) + { + /* Try to free memory from tmem. */ + pg = tmem_relinquish_pages(order, memflags); + spin_unlock(&heap_lock); + return pg; + } + + pg = get_free_buddy(zone_lo, zone_hi, order, memflags, d); + if ( !pg ) + { + /* No suitable memory blocks. Fail the request. */ + spin_unlock(&heap_lock); + return NULL; } - not_found: - /* No suitable memory blocks. Fail the request. */ - spin_unlock(&heap_lock); - return NULL; - - found: + node = phys_to_nid(page_to_maddr(pg)); + zone = page_to_zone(pg); + buddy_order = PFN_ORDER(pg); if ( pg->u.free.first_dirty != INVALID_DIRTY_IDX ) first_dirty_pg = pg + pg->u.free.first_dirty; - - /* We may have to halve the chunk a number of times. */ - while ( j != order ) + + /* We may have to halve the chunk a number of times. */ + while ( buddy_order != order ) { unsigned int first_dirty; - if ( first_dirty_pg && ((pg + (1 << j)) > first_dirty_pg) ) + if ( first_dirty_pg && ((pg + (1 << buddy_order)) > first_dirty_pg) ) { if ( pg < first_dirty_pg ) first_dirty = (first_dirty_pg - pg) / sizeof(*pg); @@ -833,8 +848,8 @@ static struct page_info *alloc_heap_pages( else first_dirty = INVALID_DIRTY_IDX; - page_list_add_scrub(pg, node, zone, --j, first_dirty); - pg += 1 << j; + page_list_add_scrub(pg, node, zone, --buddy_order, first_dirty); + pg += 1 << buddy_order; } ASSERT(avail[node][zone] >= request);