From patchwork Fri May 19 15:50:35 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Boris Ostrovsky X-Patchwork-Id: 9737505 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 060906020B for ; Fri, 19 May 2017 15:52:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 07C5B26AE3 for ; Fri, 19 May 2017 15:52:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F0A3628179; Fri, 19 May 2017 15:52:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 8202726AE3 for ; Fri, 19 May 2017 15:52:23 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dBkA4-0000ks-Me; Fri, 19 May 2017 15:49:52 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dBkA3-0000f0-WA for xen-devel@lists.xen.org; Fri, 19 May 2017 15:49:52 +0000 Received: from [85.158.139.211] by server-12.bemta-5.messagelabs.com id 27/C4-01735-F141F195; Fri, 19 May 2017 15:49:51 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupkkeJIrShJLcpLzFFi42LpnVTnqisjIh9 pcPOloMWSj4tZHBg9ju7+zRTAGMWamZeUX5HAmvFrhkLBFbWKWYcOsjYwtsp0MXJxCAl0MElc fXeDEcL5xijRsOsxG4SzgVHi/8GrzBBOD6PExdWLgBxODjYBI4mzR6czgtgiAtIS1z5fBmtnF mhgknh+7iBYQljAXeJqYxMbiM0ioCoxsWceWDOvgJfEsRlrwOISAgoSUx6+B4tzCnhLvJk0ix XEFgKq+TWrjxWixliib1YfywRGvgWMDKsYNYpTi8pSi3SNDPSSijLTM0pyEzNzdA0NTPVyU4u LE9NTcxKTivWS83M3MQKDpZ6BgXEHY+Nsv0OMkhxMSqK8joflIoX4kvJTKjMSizPii0pzUosP McpwcChJ8J4Tko8UEixKTU+tSMvMAYYtTFqCg0dJhPcISJq3uCAxtzgzHSJ1ilGX493SD++Zh Fjy8vNSpcR5s0CKBECKMkrz4EbAYugSo6yUMC8jAwODEE9BalFuZgmq/CtGcQ5GJWHeBJApPJ l5JXCbXgEdwQR0RPMDaZAjShIRUlINjFwPDMot83pkRJhKzfsfzym5X5ui6pSp2/2v8aS4/lS BLqHbWYX/o91Fvp51/bP7+k1fjbk2/a2JNy/csn7wqyeOT3PG7W1X89hmbQxSizSo3RliP21J xqtlPi1y5zsdr8re4b+zceneMus8r0MxOTlPHWwUe7cL5c5t3GTrzF1U7S7w546TEktxRqKhF nNRcSIAQRCy9pwCAAA= X-Env-Sender: boris.ostrovsky@oracle.com X-Msg-Ref: server-10.tower-206.messagelabs.com!1495208986!77899637!1 X-Originating-IP: [141.146.126.69] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTQxLjE0Ni4xMjYuNjkgPT4gMjc3MjE4\n X-StarScan-Received: X-StarScan-Version: 9.4.12; banners=-,-,- X-VirusChecked: Checked Received: (qmail 64276 invoked from network); 19 May 2017 15:49:47 -0000 Received: from aserp1040.oracle.com (HELO aserp1040.oracle.com) (141.146.126.69) by server-10.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 19 May 2017 15:49:47 -0000 Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by aserp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id v4JFncjG027845 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 19 May 2017 15:49:39 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserv0021.oracle.com (8.13.8/8.14.4) with ESMTP id v4JFncpC030281 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Fri, 19 May 2017 15:49:38 GMT Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id v4JFncVd030758; Fri, 19 May 2017 15:49:38 GMT Received: from ovs104.us.oracle.com (/10.149.76.204) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Fri, 19 May 2017 08:49:38 -0700 From: Boris Ostrovsky To: xen-devel@lists.xen.org Date: Fri, 19 May 2017 11:50:35 -0400 Message-Id: <1495209040-11101-4-git-send-email-boris.ostrovsky@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1495209040-11101-1-git-send-email-boris.ostrovsky@oracle.com> References: <1495209040-11101-1-git-send-email-boris.ostrovsky@oracle.com> X-Source-IP: aserv0021.oracle.com [141.146.126.233] Cc: sstabellini@kernel.org, wei.liu2@citrix.com, George.Dunlap@eu.citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, tim@xen.org, jbeulich@suse.com, Boris Ostrovsky Subject: [Xen-devel] [PATCH v4 3/8] mm: Scrub pages in alloc_heap_pages() if needed X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP When allocating pages in alloc_heap_pages() first look for clean pages. If none is found then retry, take pages marked as unscrubbed and scrub them. Note that we shouldn't find unscrubbed pages in alloc_heap_pages() yet. However, this will become possible when we stop scrubbing from free_heap_pages() and instead do it from idle loop. Since not all allocations require clean pages (such as xenheap allocations) introduce MEMF_no_scrub flag that callers can set if they are willing to consume unscrubbed pages. Signed-off-by: Boris Ostrovsky --- Changes in v4: * Add MEMF_no_tlbflush flag xen/common/page_alloc.c | 43 ++++++++++++++++++++++++++++++++++++------- xen/include/xen/mm.h | 2 ++ 2 files changed, 38 insertions(+), 7 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 1e57885..b7c7426 100644 --- a/xen/common/page_alloc.c +++ b/xen/common/page_alloc.c @@ -703,6 +703,7 @@ static struct page_info *get_free_buddy(unsigned int zone_lo, nodemask_t nodemask = d ? d->node_affinity : node_online_map; unsigned int j, zone, nodemask_retry = 0, request = 1UL << order; struct page_info *pg; + bool use_unscrubbed = (memflags & MEMF_no_scrub); if ( node == NUMA_NO_NODE ) { @@ -734,8 +735,15 @@ static struct page_info *get_free_buddy(unsigned int zone_lo, /* Find smallest order which can satisfy the request. */ for ( j = order; j <= MAX_ORDER; j++ ) + { if ( (pg = page_list_remove_head(&heap(node, zone, j))) ) - return pg; + { + if ( (order == 0) || use_unscrubbed || + pg->u.free.first_dirty == INVALID_DIRTY_IDX) + return pg; + page_list_add_tail(pg, &heap(node, zone, j)); + } + } } while ( zone-- > zone_lo ); /* careful: unsigned zone may wrap */ if ( (memflags & MEMF_exact_node) && req_node != NUMA_NO_NODE ) @@ -821,9 +829,16 @@ static struct page_info *alloc_heap_pages( pg = get_free_buddy(zone_lo, zone_hi, order, memflags, d); if ( !pg ) { - /* No suitable memory blocks. Fail the request. */ - spin_unlock(&heap_lock); - return NULL; + /* Try now getting a dirty buddy. */ + if ( !(memflags & MEMF_no_scrub) ) + pg = get_free_buddy(zone_lo, zone_hi, order, + memflags | MEMF_no_scrub, d); + if ( !pg ) + { + /* No suitable memory blocks. Fail the request. */ + spin_unlock(&heap_lock); + return NULL; + } } node = phys_to_nid(page_to_maddr(pg)); @@ -855,10 +870,24 @@ static struct page_info *alloc_heap_pages( if ( d != NULL ) d->last_alloc_node = node; + need_scrub &= !(memflags & MEMF_no_scrub); for ( i = 0; i < (1 << order); i++ ) { /* Reference count must continuously be zero for free pages. */ - BUG_ON(pg[i].count_info != PGC_state_free); + BUG_ON((pg[i].count_info & ~PGC_need_scrub ) != PGC_state_free); + + if ( test_bit(_PGC_need_scrub, &pg[i].count_info) ) + { + if ( need_scrub ) + scrub_one_page(&pg[i]); + node_need_scrub[node]--; + /* + * Technically, we need to set first_dirty to INVALID_DIRTY_IDX + * on buddy's head. However, since we assign pg[i].count_info + * below, we can skip this. + */ + } + pg[i].count_info = PGC_state_inuse; if ( !(memflags & MEMF_no_tlbflush) ) @@ -1737,7 +1766,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags) ASSERT(!in_irq()); pg = alloc_heap_pages(MEMZONE_XEN, MEMZONE_XEN, - order, memflags, NULL); + order, memflags | MEMF_no_scrub, NULL); if ( unlikely(pg == NULL) ) return NULL; @@ -1787,7 +1816,7 @@ void *alloc_xenheap_pages(unsigned int order, unsigned int memflags) if ( !(memflags >> _MEMF_bits) ) memflags |= MEMF_bits(xenheap_bits); - pg = alloc_domheap_pages(NULL, order, memflags); + pg = alloc_domheap_pages(NULL, order, memflags | MEMF_no_scrub); if ( unlikely(pg == NULL) ) return NULL; diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h index 88de3c1..0d4b7c2 100644 --- a/xen/include/xen/mm.h +++ b/xen/include/xen/mm.h @@ -224,6 +224,8 @@ struct npfec { #define MEMF_no_owner (1U<<_MEMF_no_owner) #define _MEMF_no_tlbflush 6 #define MEMF_no_tlbflush (1U<<_MEMF_no_tlbflush) +#define _MEMF_no_scrub 7 +#define MEMF_no_scrub (1U<<_MEMF_no_scrub) #define _MEMF_node 8 #define MEMF_node_mask ((1U << (8 * sizeof(nodeid_t))) - 1) #define MEMF_node(n) ((((n) + 1) & MEMF_node_mask) << _MEMF_node)