From patchwork Thu Jun 22 18:57:05 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Ostrovsky X-Patchwork-Id: 9805101 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id A037260329 for ; Thu, 22 Jun 2017 18:57:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 90EBE283DA for ; Thu, 22 Jun 2017 18:57:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 85360286E1; Thu, 22 Jun 2017 18:57:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id C85E5283DA for ; Thu, 22 Jun 2017 18:57:52 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dO7Ga-0005SI-3L; Thu, 22 Jun 2017 18:55:44 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dO7GY-0005QE-Tt for xen-devel@lists.xen.org; Thu, 22 Jun 2017 18:55:43 +0000 Received: from [193.109.254.147] by server-1.bemta-6.messagelabs.com id F0/7E-06719-EA21C495; Thu, 22 Jun 2017 18:55:42 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupkkeJIrShJLcpLzFFi42KZM10+UHetkE+ kwdQ3chZLPi5mcWD0OLr7N1MAYxRrZl5SfkUCa8a5/SwFL7Uq7jYsYW9gPCPfxcjFISTQwSTR +eAqO4TzhVGidXMPM4SzkVHi5umTUE4Po8SljXuBHE4ONgEjibNHpzOC2CIC0hLXPl9mBCliF mhgknh+7iBYQljAXWLPit1sIDaLgKrE2xXvwWxeAS+Jl9uXsoPYEgIKElMevgcbyingLfH41F uwXiGgmqVv10LVGEv0zepjmcDIt4CRYRWjRnFqUVlqka6hgV5SUWZ6RkluYmYOkGeml5taXJy YnpqTmFSsl5yfu4kRGCwMQLCD8d6ygEOMkhxMSqK8m4u9I4X4kvJTKjMSizPii0pzUosPMcpw cChJ8K4W9IkUEixKTU+tSMvMAYYtTFqCg0dJhPcLH1Cat7ggMbc4Mx0idYpRl2PD6vVfmIRY8 vLzUqXEefeBzBAAKcoozYMbAYuhS4yyUsK8jEBHCfEUpBblZpagyr9iFOdgVBLmFRcAmsKTmV cCt+kV0BFMQEe8OOIBckRJIkJKqoFxsdtXljsnvi347id56MWSoPmud2oipDmXyZX//OIX+Or +fYNrGS8N67f5zVF7c2d9/2X2q4k3gl6vl3my7v3el4uZLL3OX7W+t0LyAv/Hm95TX+1j/Gjg KbzpxjfunCbme3fuKSRvaGZstXbZHOfK7namyFRC+q7xLe8TEirqFQWeN5dItntbKbEUZyQaa jEXFScCAOWzsU2cAgAA X-Env-Sender: boris.ostrovsky@oracle.com X-Msg-Ref: server-2.tower-27.messagelabs.com!1498157739!49482737!1 X-Originating-IP: [156.151.31.81] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTU2LjE1MS4zMS44MSA9PiAyODgzMzk=\n X-StarScan-Received: X-StarScan-Version: 9.4.19; banners=-,-,- X-VirusChecked: Checked Received: (qmail 36902 invoked from network); 22 Jun 2017 18:55:41 -0000 Received: from userp1040.oracle.com (HELO userp1040.oracle.com) (156.151.31.81) by server-2.tower-27.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 22 Jun 2017 18:55:41 -0000 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v5MItWhn011642 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 22 Jun 2017 18:55:32 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by aserv0021.oracle.com (8.13.8/8.14.4) with ESMTP id v5MItVrW020144 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 22 Jun 2017 18:55:32 GMT Received: from abhmp0015.oracle.com (abhmp0015.oracle.com [141.146.116.21]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id v5MItVGP005582; Thu, 22 Jun 2017 18:55:31 GMT Received: from ovs104.us.oracle.com (/10.149.76.204) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Thu, 22 Jun 2017 11:55:31 -0700 From: Boris Ostrovsky To: xen-devel@lists.xen.org Date: Thu, 22 Jun 2017 14:57:05 -0400 Message-Id: <1498157830-21845-4-git-send-email-boris.ostrovsky@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1498157830-21845-1-git-send-email-boris.ostrovsky@oracle.com> References: <1498157830-21845-1-git-send-email-boris.ostrovsky@oracle.com> X-Source-IP: aserv0021.oracle.com [141.146.126.233] Cc: sstabellini@kernel.org, wei.liu2@citrix.com, George.Dunlap@eu.citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, tim@xen.org, jbeulich@suse.com, Boris Ostrovsky Subject: [Xen-devel] [PATCH v5 3/8] mm: Scrub pages in alloc_heap_pages() if needed X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP When allocating pages in alloc_heap_pages() first look for clean pages. If none is found then retry, take pages marked as unscrubbed and scrub them. Note that we shouldn't find unscrubbed pages in alloc_heap_pages() yet. However, this will become possible when we stop scrubbing from free_heap_pages() and instead do it from idle loop. Since not all allocations require clean pages (such as xenheap allocations) introduce MEMF_no_scrub flag that callers can set if they are willing to consume unscrubbed pages. Signed-off-by: Boris Ostrovsky Reviewed-by: Jan Beulich --- Changes in v5: * Added comment explaining why we always grab order 0 pages in alloc_heap_pages) * Dropped the somewhat confusing comment about not needing to set first_dirty in alloc_heap_pages(). * Moved first bit of _MEMF_node by 8 to accommodate MEMF_no_scrub (bit 7 is no longer available) xen/common/page_alloc.c | 36 +++++++++++++++++++++++++++++++----- xen/include/xen/mm.h | 4 +++- 2 files changed, 34 insertions(+), 6 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 89fe3ce..9aac196 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -703,6 +703,7 @@ static struct page_info *get_free_buddy(unsigned int zone_lo, nodemask_t nodemask = d ? d->node_affinity : node_online_map; unsigned int j, zone, nodemask_retry = 0; struct page_info *pg; + bool use_unscrubbed = (memflags & MEMF_no_scrub); if ( node == NUMA_NO_NODE ) { @@ -734,8 +735,20 @@ static struct page_info *get_free_buddy(unsigned int zone_lo, /* Find smallest order which can satisfy the request. */ for ( j = order; j <= MAX_ORDER; j++ ) + { if ( (pg = page_list_remove_head(&heap(node, zone, j))) ) - return pg; + { + /* + * We grab single pages (order=0) even if they are + * unscrubbed. Given that scrubbing one page is fairly quick + * it is not worth breaking higher orders. + */ + if ( (order == 0) || use_unscrubbed || + pg->u.free.first_dirty == INVALID_DIRTY_IDX) + return pg; + page_list_add_tail(pg, &heap(node, zone, j)); + } + } } while ( zone-- > zone_lo ); /* careful: unsigned zone may wrap */ if ( (memflags & MEMF_exact_node) && req_node != NUMA_NO_NODE ) @@ -775,7 +788,7 @@ static struct page_info *alloc_heap_pages( unsigned int i, buddy_order, zone; unsigned long request = 1UL << order; struct page_info *pg, *first_dirty_pg = NULL; - bool_t need_tlbflush = 0; + bool need_scrub, need_tlbflush = false; uint32_t tlbflush_timestamp = 0; /* Make sure there are enough bits in memflags for nodeID. */ @@ -819,6 +832,10 @@ static struct page_info *alloc_heap_pages( } pg = get_free_buddy(zone_lo, zone_hi, order, memflags, d); + /* Try getting a dirty buddy if we couldn't get a clean one. */ + if ( !pg && !(memflags & MEMF_no_scrub) ) + pg = get_free_buddy(zone_lo, zone_hi, order, + memflags | MEMF_no_scrub, d); if ( !pg ) { /* No suitable memory blocks. Fail the request. */ @@ -862,10 +879,19 @@ static struct page_info *alloc_heap_pages( if ( d != NULL ) d->last_alloc_node = node; + need_scrub = !!first_dirty_pg && !(memflags & MEMF_no_scrub); for ( i = 0; i < (1 << order); i++ ) { /* Reference count must continuously be zero for free pages. */ - BUG_ON(pg[i].count_info != PGC_state_free); + BUG_ON((pg[i].count_info & ~PGC_need_scrub) != PGC_state_free); + + if ( test_bit(_PGC_need_scrub, &pg[i].count_info) ) + { + if ( need_scrub ) + scrub_one_page(&pg[i]); + node_need_scrub[node]--; + } + pg[i].count_info = PGC_state_inuse; if ( !(memflags & MEMF_no_tlbflush) ) @@ -1749,7 +1775,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags) ASSERT(!in_irq()); pg = alloc_heap_pages(MEMZONE_XEN, MEMZONE_XEN, - order, memflags, NULL); + order, memflags | MEMF_no_scrub, NULL); if ( unlikely(pg == NULL) ) return NULL; @@ -1799,7 +1825,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags) if ( !(memflags >> _MEMF_bits) ) memflags |= MEMF_bits(xenheap_bits); - pg = alloc_domheap_pages(NULL, order, memflags); + pg = alloc_domheap_pages(NULL, order, memflags | MEMF_no_scrub); if ( unlikely(pg == NULL) ) return NULL; diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h index 3d3f31b..5f3d84a 100644 --- a/xen/include/xen/mm.h +++ b/xen/include/xen/mm.h @@ -238,7 +238,9 @@ struct npfec { #define MEMF_no_tlbflush (1U<<_MEMF_no_tlbflush) #define _MEMF_no_icache_flush 7 #define MEMF_no_icache_flush (1U<<_MEMF_no_icache_flush) -#define _MEMF_node 8 +#define _MEMF_no_scrub 8 +#define MEMF_no_scrub (1U<<_MEMF_no_scrub) +#define _MEMF_node 16 #define MEMF_node_mask ((1U << (8 * sizeof(nodeid_t))) - 1) #define MEMF_node(n) ((((n) + 1) & MEMF_node_mask) << _MEMF_node) #define MEMF_get_node(f) ((((f) >> _MEMF_node) - 1) & MEMF_node_mask)