From patchwork Fri Jul 15 09:45:11 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Lieven X-Patchwork-Id: 9231465 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3624860574 for ; Fri, 15 Jul 2016 09:45:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 263CA1FE95 for ; Fri, 15 Jul 2016 09:45:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1ACEA27FB7; Fri, 15 Jul 2016 09:45:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 94AB71FE95 for ; Fri, 15 Jul 2016 09:45:49 +0000 (UTC) Received: from localhost ([::1]:59503 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bNzgo-0005TB-L8 for patchwork-qemu-devel@patchwork.kernel.org; Fri, 15 Jul 2016 05:45:46 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:35457) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bNzgW-0005Su-6P for qemu-devel@nongnu.org; Fri, 15 Jul 2016 05:45:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bNzgS-00053m-0Q for qemu-devel@nongnu.org; Fri, 15 Jul 2016 05:45:27 -0400 Received: from mx-v6.kamp.de ([2a02:248:0:51::16]:39351 helo=mx01.kamp.de) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bNzgR-00053O-N4 for qemu-devel@nongnu.org; Fri, 15 Jul 2016 05:45:23 -0400 Received: (qmail 14961 invoked by uid 89); 15 Jul 2016 09:45:20 -0000 Received: from [195.62.97.28] by client-16-kamp (envelope-from , uid 89) with qmail-scanner-2010/03/19-MF (clamdscan: 0.99.2/21906. hbedv: 8.3.40.44/7.12.99.34. avast: 1.2.2/16071500. spamassassin: 3.4.1. Clear:RC:1(195.62.97.28):. Processed in 0.062631 secs); 15 Jul 2016 09:45:20 -0000 Received: from smtp.kamp.de (HELO submission.kamp.de) ([195.62.97.28]) by mx01.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted); 15 Jul 2016 09:45:20 -0000 X-GL_Whitelist: yes Received: (qmail 15896 invoked from network); 15 Jul 2016 09:45:19 -0000 Received: from lieven-pc.kamp-intra.net (HELO lieven-pc) (relay@kamp.de@::ffff:172.21.12.60) by submission.kamp.de with ESMTPS (DHE-RSA-AES256-GCM-SHA384 encrypted) ESMTPA; 15 Jul 2016 09:45:19 -0000 Received: by lieven-pc (Postfix, from userid 1000) id EDB84207EF; Fri, 15 Jul 2016 11:45:14 +0200 (CEST) From: Peter Lieven To: qemu-devel@nongnu.org Date: Fri, 15 Jul 2016 11:45:11 +0200 Message-Id: <1468575911-20656-1-git-send-email-pl@kamp.de> X-Mailer: git-send-email 1.9.1 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2a02:248:0:51::16 Subject: [Qemu-devel] [PATCH] vnc-tight: fix regression with libxenstore X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: jgross@suse.com, pbonzini@redhat.com, Peter Lieven , kraxel@redhat.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP commit 095497ff added thread local storage for the color counting palette. Unfortunately, a VncPalette is about 7kB on a x86_64 system. This memory is reserved from the stack of every thread and it exhausted the stack space of a libxenstore thread. Fix this by allocating memory only for the VNC encoding thread. Fixes: 095497ffc66b7f031ff2a17f1e50f5cb105ce588 Reported-by: Juergen Gross Tested-by: Juergen Gross Signed-off-by: Peter Lieven --- ui/vnc-enc-tight.c | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/ui/vnc-enc-tight.c b/ui/vnc-enc-tight.c index b8581dd..2b58739 100644 --- a/ui/vnc-enc-tight.c +++ b/ui/vnc-enc-tight.c @@ -1457,11 +1457,17 @@ static int send_sub_rect_jpeg(VncState *vs, int x, int y, int w, int h, } #endif -static __thread VncPalette color_count_palette; +static __thread VncPalette *color_count_palette; +static __thread Notifier vnc_tight_cleanup_notifier; + +static void vnc_tight_cleanup(Notifier *n, void *value) +{ + g_free(color_count_palette); + color_count_palette = NULL; +} static int send_sub_rect(VncState *vs, int x, int y, int w, int h) { - VncPalette *palette = &color_count_palette; uint32_t bg = 0, fg = 0; int colors; int ret = 0; @@ -1470,6 +1476,12 @@ static int send_sub_rect(VncState *vs, int x, int y, int w, int h) bool allow_jpeg = true; #endif + if (!color_count_palette) { + color_count_palette = g_malloc(sizeof(VncPalette)); + vnc_tight_cleanup_notifier.notify = vnc_tight_cleanup; + qemu_thread_atexit_add(&vnc_tight_cleanup_notifier); + } + vnc_framebuffer_update(vs, x, y, w, h, vs->tight.type); vnc_tight_start(vs); @@ -1490,17 +1502,19 @@ static int send_sub_rect(VncState *vs, int x, int y, int w, int h) } #endif - colors = tight_fill_palette(vs, x, y, w * h, &bg, &fg, palette); + colors = tight_fill_palette(vs, x, y, w * h, &bg, &fg, color_count_palette); #ifdef CONFIG_VNC_JPEG if (allow_jpeg && vs->tight.quality != (uint8_t)-1) { - ret = send_sub_rect_jpeg(vs, x, y, w, h, bg, fg, colors, palette, - force_jpeg); + ret = send_sub_rect_jpeg(vs, x, y, w, h, bg, fg, colors, + color_count_palette, force_jpeg); } else { - ret = send_sub_rect_nojpeg(vs, x, y, w, h, bg, fg, colors, palette); + ret = send_sub_rect_nojpeg(vs, x, y, w, h, bg, fg, colors, + color_count_palette); } #else - ret = send_sub_rect_nojpeg(vs, x, y, w, h, bg, fg, colors, palette); + ret = send_sub_rect_nojpeg(vs, x, y, w, h, bg, fg, colors, + color_count_palette); #endif return ret;