From patchwork Wed Aug 16 18:33:09 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Ostrovsky X-Patchwork-Id: 9904481 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B344C600CA for ; Wed, 16 Aug 2017 18:33:09 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A6F4428A40 for ; Wed, 16 Aug 2017 18:33:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9B82828A4F; Wed, 16 Aug 2017 18:33:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8254628A41 for ; Wed, 16 Aug 2017 18:33:07 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1di35c-0005Sn-U0; Wed, 16 Aug 2017 18:30:48 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1di35b-0005SJ-CJ for xen-devel@lists.xen.org; Wed, 16 Aug 2017 18:30:47 +0000 Received: from [85.158.137.68] by server-17.bemta-3.messagelabs.com id 76/62-01859-55F84995; Wed, 16 Aug 2017 18:30:45 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupnkeJIrShJLcpLzFFi42LpnVTnqhvaPyX SYN9lGYslHxezODB6HN39mymAMYo1My8pvyKBNePY0glsBb9UKi6+6GRqYOyQ7mLk4hASmMwk cfPUQzYI5zejxOztZ9khnA2MEkce3GKBcHoYJd5uOABUxsnBJmAkcfbodEYQW0RAWuLa58uMI EXMAtOYJPZ+eweWEBbwlHjdsoMdxGYRUJU4unQCkM3BwQsU//CFCyQsIaAgMeXhe2YQm1PAS2 Lt9nVg84WASpZ8OMgMUWMs0f72ItsERr4FjAyrGDWKU4vKUot0DQ31kooy0zNKchMzc3QNDYz 1clOLixPTU3MSk4r1kvNzNzECg4UBCHYwrv7tdIhRkoNJSZTXK39KpBBfUn5KZUZicUZ8UWlO avEhRhkODiUJ3vxeoJxgUWp6akVaZg4wbGHSEhw8SiK8M3qA0rzFBYm5xZnpEKlTjMYcG1av/ 8LE8WrC/29MQix5+XmpUuK8b0EmCYCUZpTmwQ2CxdMlRlkpYV5GoNOEeApSi3IzS1DlXzGKcz AqCfOeBJnCk5lXArfvFdApTECnXGmfBHJKSSJCSqqBkeWGSNiSBfqBrB8nm7avdRE/aia9+kV u3u/P2yRXS3NWxM3wdLvImBF/597TjUutWSa7aHHuZslVvfopeMek0lk/slVWWGamyuaaJaom FPh3LuE4t2RmkKLhM89Ppg4ZhXE+KqWPb9VtYQgqs+u9d8ueJ/lBmYNsulvSh1wzrprp9cdab BOUWIozEg21mIuKEwHPFCL/ogIAAA== X-Env-Sender: boris.ostrovsky@oracle.com X-Msg-Ref: server-4.tower-31.messagelabs.com!1502908244!52240719!1 X-Originating-IP: [141.146.126.69] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTQxLjE0Ni4xMjYuNjkgPT4gMjc3MjE4\n X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 13026 invoked from network); 16 Aug 2017 18:30:45 -0000 Received: from aserp1040.oracle.com (HELO aserp1040.oracle.com) (141.146.126.69) by server-4.tower-31.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 16 Aug 2017 18:30:45 -0000 Received: from aserv0022.oracle.com (aserv0022.oracle.com [141.146.126.234]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v7GIUaCI003168 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 16 Aug 2017 18:30:37 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserv0022.oracle.com (8.14.4/8.14.4) with ESMTP id v7GIUa8B010734 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 16 Aug 2017 18:30:36 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id v7GIUZAP003322; Wed, 16 Aug 2017 18:30:35 GMT Received: from ovs104.us.oracle.com (/10.149.76.204) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Wed, 16 Aug 2017 11:30:35 -0700 From: Boris Ostrovsky To: xen-devel@lists.xen.org Date: Wed, 16 Aug 2017 14:33:09 -0400 Message-Id: <1502908394-9760-4-git-send-email-boris.ostrovsky@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1502908394-9760-1-git-send-email-boris.ostrovsky@oracle.com> References: <1502908394-9760-1-git-send-email-boris.ostrovsky@oracle.com> X-Source-IP: aserv0022.oracle.com [141.146.126.234] Cc: sstabellini@kernel.org, wei.liu2@citrix.com, George.Dunlap@eu.citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, tim@xen.org, julien.grall@arm.com, jbeulich@suse.com, Boris Ostrovsky Subject: [Xen-devel] [PATCHES v8 3/8] mm: Scrub pages in alloc_heap_pages() if needed X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP When allocating pages in alloc_heap_pages() first look for clean pages. If none is found then retry, take pages marked as unscrubbed and scrub them. Note that we shouldn't find unscrubbed pages in alloc_heap_pages() yet. However, this will become possible when we stop scrubbing from free_heap_pages() and instead do it from idle loop. Since not all allocations require clean pages (such as xenheap allocations) introduce MEMF_no_scrub flag that callers can set if they are willing to consume unscrubbed pages. Signed-off-by: Boris Ostrovsky Reviewed-by: Jan Beulich --- xen/common/page_alloc.c | 33 +++++++++++++++++++++++++++++---- xen/include/xen/mm.h | 4 +++- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 5c550b5..1303736 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -702,6 +702,7 @@ static struct page_info *get_free_buddy(unsigned int zone_lo, nodemask_t nodemask = d ? d->node_affinity : node_online_map; unsigned int j, zone, nodemask_retry = 0; struct page_info *pg; + bool use_unscrubbed = (memflags & MEMF_no_scrub); if ( node == NUMA_NO_NODE ) { @@ -733,8 +734,20 @@ static struct page_info *get_free_buddy(unsigned int zone_lo, /* Find smallest order which can satisfy the request. */ for ( j = order; j <= MAX_ORDER; j++ ) + { if ( (pg = page_list_remove_head(&heap(node, zone, j))) ) - return pg; + { + /* + * We grab single pages (order=0) even if they are + * unscrubbed. Given that scrubbing one page is fairly quick + * it is not worth breaking higher orders. + */ + if ( (order == 0) || use_unscrubbed || + pg->u.free.first_dirty == INVALID_DIRTY_IDX) + return pg; + page_list_add_tail(pg, &heap(node, zone, j)); + } + } } while ( zone-- > zone_lo ); /* careful: unsigned zone may wrap */ if ( (memflags & MEMF_exact_node) && req_node != NUMA_NO_NODE ) @@ -818,6 +831,10 @@ static struct page_info *alloc_heap_pages( } pg = get_free_buddy(zone_lo, zone_hi, order, memflags, d); + /* Try getting a dirty buddy if we couldn't get a clean one. */ + if ( !pg && !(memflags & MEMF_no_scrub) ) + pg = get_free_buddy(zone_lo, zone_hi, order, + memflags | MEMF_no_scrub, d); if ( !pg ) { /* No suitable memory blocks. Fail the request. */ @@ -863,7 +880,15 @@ static struct page_info *alloc_heap_pages( for ( i = 0; i < (1 << order); i++ ) { /* Reference count must continuously be zero for free pages. */ - BUG_ON(pg[i].count_info != PGC_state_free); + BUG_ON((pg[i].count_info & ~PGC_need_scrub) != PGC_state_free); + + if ( test_bit(_PGC_need_scrub, &pg[i].count_info) ) + { + if ( !(memflags & MEMF_no_scrub) ) + scrub_one_page(&pg[i]); + node_need_scrub[node]--; + } + pg[i].count_info = PGC_state_inuse; if ( !(memflags & MEMF_no_tlbflush) ) @@ -1737,7 +1762,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags) ASSERT(!in_irq()); pg = alloc_heap_pages(MEMZONE_XEN, MEMZONE_XEN, - order, memflags, NULL); + order, memflags | MEMF_no_scrub, NULL); if ( unlikely(pg == NULL) ) return NULL; @@ -1787,7 +1812,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags) if ( !(memflags >> _MEMF_bits) ) memflags |= MEMF_bits(xenheap_bits); - pg = alloc_domheap_pages(NULL, order, memflags); + pg = alloc_domheap_pages(NULL, order, memflags | MEMF_no_scrub); if ( unlikely(pg == NULL) ) return NULL; diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h index 503b92e..e1f9c42 100644 --- a/xen/include/xen/mm.h +++ b/xen/include/xen/mm.h @@ -248,7 +248,9 @@ struct npfec { #define MEMF_no_tlbflush (1U<<_MEMF_no_tlbflush) #define _MEMF_no_icache_flush 7 #define MEMF_no_icache_flush (1U<<_MEMF_no_icache_flush) -#define _MEMF_node 8 +#define _MEMF_no_scrub 8 +#define MEMF_no_scrub (1U<<_MEMF_no_scrub) +#define _MEMF_node 16 #define MEMF_node_mask ((1U << (8 * sizeof(nodeid_t))) - 1) #define MEMF_node(n) ((((n) + 1) & MEMF_node_mask) << _MEMF_node) #define MEMF_get_node(f) ((((f) >> _MEMF_node) - 1) & MEMF_node_mask)