From patchwork Wed Jun 21 07:20:20 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Schmoll X-Patchwork-Id: 9800981 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3FE5C600C5 for ; Wed, 21 Jun 2017 07:23:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6DBE528539 for ; Wed, 21 Jun 2017 07:23:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 61F2328548; Wed, 21 Jun 2017 07:23:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_MED, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 3BF2428539 for ; Wed, 21 Jun 2017 07:23:08 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dNZwf-00052I-3U; Wed, 21 Jun 2017 07:20:57 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dNZwe-00051y-13 for xen-devel@lists.xenproject.org; Wed, 21 Jun 2017 07:20:56 +0000 Received: from [193.109.254.147] by server-6.bemta-6.messagelabs.com id F6/0B-03920-75E1A495; Wed, 21 Jun 2017 07:20:55 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprCIsWRWlGSWpSXmKPExsVyMbThkG6YnFe kQcdzJYvvWyYzOTB6HP5whSWAMYo1My8pvyKBNePVzM3sBeuiKy4euM/SwHjXs4uRi0NIYCaj xPfJf5hBHBaBSywSPb+eMYI4EgLvWCTuv+4FcjiBnCSJ6bu2M0PY1RK7rj0AiwsJaEp86e5ih Bi1gEli8ec/bCAJNgEDidkrL4I1iAgoSdxbNZkJxGYWcJHomtcMViMs4CFx/84esEEsAqoSR0 7tYgexeQXsJN4enQK1TF5iV9tF1gmMfAsYGVYxqhenFpWlFuka6yUVZaZnlOQmZuboGhqY6eW mFhcnpqfmJCYV6yXn525iBIYKAxDsYOz453SIUZKDSUmUt+SLZ6QQX1J+SmVGYnFGfFFpTmrx IUYZDg4lCd4Zsl6RQoJFqempFWmZOcCghUlLcPAoifBu4QdK8xYXJOYWZ6ZDpE4xWnJcubLuC xPHhtXrgeSUA9u/MAmx5OXnpUqJ8xaCzBMAacgozYMbB4usS4yyUsK8jEAHCvEUpBblZpagyr 9iFOdgVBLmTZMBmsKTmVcCt/UV0EFMQAe9OOIBclBJIkJKqoExIy1dcFZox87iXuvlcwXv9uz YeWivSuz8Dfac/qqBDH2XPbW5VWZ2atz2vGzdeZvjr1LB2Rc/n9xo3tTbVWLHJn5l/xKOuhzO r7cuP139pMfi7a0tC9b9zS0SW+TUvcI/WYJ5vvCzabEPEsvDTe4/3Z12b/ul1wXab3dNeZZze Pms3fyWts+XK7EUZyQaajEXFScCALbVauinAgAA X-Env-Sender: eggi.innovations@gmail.com X-Msg-Ref: server-5.tower-27.messagelabs.com!1498029654!103216453!1 X-Originating-IP: [209.85.128.194] X-SpamReason: No, hits=0.0 required=7.0 tests= X-StarScan-Received: X-StarScan-Version: 9.4.19; banners=-,-,- X-VirusChecked: Checked Received: (qmail 21179 invoked from network); 21 Jun 2017 07:20:54 -0000 Received: from mail-wr0-f194.google.com (HELO mail-wr0-f194.google.com) (209.85.128.194) by server-5.tower-27.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 21 Jun 2017 07:20:54 -0000 Received: by mail-wr0-f194.google.com with SMTP id z45so23990519wrb.2 for ; Wed, 21 Jun 2017 00:20:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=uaZ5Ja/f8lh2ceQRkbZcLkjOQAdhMrjutk1CZ68hH2Q=; b=lpeypqa8On/ClH+fUAZgCVve7G9BrxOYAoIVdna4vbMfI3StAFJwBWxmwZpkoP/h93 hpDIOnuq2xl9nBYrFMG+GsoeRIMgnuPn/lzPI8n9NxeH6DHjdVP54hsPRc5pCIiavZoC XmndUBRdvmSxgS2ZFJ/oZp18TgLuzHCuU57U+WMaFEhB9GyD0nViYPBII/rqiII9QbQA pPqeWFCV3pDyMpbe9OjvlkHxVYZlK9s2kkqmEldsAhDEsoXqnXN0MC7528X8Hai9Q4Rf 9tatm+26y1Cq8lvzgjhY7CsSm0MP02AzBaQ15pcdfyXUrNOfowQ4cvendHjNvPeLZU8d IIWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=uaZ5Ja/f8lh2ceQRkbZcLkjOQAdhMrjutk1CZ68hH2Q=; b=imaz00Ow6BzFoREGW4J3NbFsA5Op2+BQ8FoZzH9tO9evgxSrTYV3XWe1GFj0cDk8A3 oM7FsYJhRVbuAuSn4zMvvyfko+LB92/2uPq2C4tnapyRI4JvdPRIHF9X0MkqSbf54rnp pXQ86NMRzYLABOx6XQHFOWSgtSmsscWgFGi+/9oBh7ePNWVp3gOp1+pwHW8qp2cXV7Zd xvo2ePREB9A0j1w3L5s872+CBwCpvmwPUOEaO89JIcx6BWg4NNj1+BFK1NpVEqLFOuTx b7z9ByfA3ufi22dsMarcXmlSd37sWeF17wQOzx0U/bon0mjYaztMFW1a9Cfu5nKawfzk 8nBA== X-Gm-Message-State: AKS2vOwKToOyV60NVlhISluTTATvAyq09qPKHD7rS+OVz5LE5ExyZHRM 1ELiY9d4rSwKZaR1iYs= X-Received: by 10.223.166.196 with SMTP id t62mr21246132wrc.52.1498029653851; Wed, 21 Jun 2017 00:20:53 -0700 (PDT) Received: from localhost.localdomain (ip-109-47-1-245.web.vodafone.de. [109.47.1.245]) by smtp.gmail.com with ESMTPSA id s95sm13838893wrc.13.2017.06.21.00.20.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Jun 2017 00:20:53 -0700 (PDT) From: Felix Schmoll To: xen-devel@lists.xenproject.org Date: Wed, 21 Jun 2017 09:20:20 +0200 Message-Id: <20170621072020.1695-1-eggi.innovations@gmail.com> X-Mailer: git-send-email 2.11.0 Cc: wei.liu2@citrix.com, Felix Schmoll Subject: [Xen-devel] [PATCH RFC] Implement hypercall for tracing of program counters X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This commit makes the changes to the hypervisor, the build system as well as libxc necessary in order to facilitate tracing of program counters. A discussion of the design can be found in the mailing list: https://lists.xen.org/archives/html/xen-devel/2017-05/threads.html#02210 The list of files to be included for tracing might still be too extensive, resulting in indeterministic tracing output for some use cases. It is also not clear what other causes of indeterminism there might be. Signed-off-by: Felix Schmoll --- tools/libxc/include/xenctrl.h | 2 ++ tools/libxc/xc_private.c | 22 +++++++++++++ tools/libxc/xc_private.h | 8 +++++ xen/Kconfig | 4 +++ xen/Rules.mk | 4 +++ xen/arch/arm/traps.c | 1 + xen/arch/x86/hvm/hypercall.c | 1 + xen/arch/x86/hypercall.c | 1 + xen/arch/x86/pv/hypercall.c | 1 + xen/common/Makefile | 13 ++++++++ xen/common/edge_trace.c | 77 +++++++++++++++++++++++++++++++++++++++++++ xen/common/edge_tracer.c | 25 ++++++++++++++ xen/include/public/xen.h | 1 + xen/include/xen/edge_trace.h | 19 +++++++++++ xen/include/xen/hypercall.h | 7 ++++ xen/include/xen/sched.h | 6 ++++ 16 files changed, 192 insertions(+) create mode 100644 xen/common/edge_trace.c create mode 100644 xen/common/edge_tracer.c create mode 100644 xen/include/xen/edge_trace.h diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h index 8c26cb4141..75e03337f9 100644 --- a/tools/libxc/include/xenctrl.h +++ b/tools/libxc/include/xenctrl.h @@ -1572,6 +1572,8 @@ int xc_domctl(xc_interface *xch, struct xen_domctl *domctl); int xc_sysctl(xc_interface *xch, struct xen_sysctl *sysctl); int xc_version(xc_interface *xch, int cmd, void *arg); +int xc_edge_trace(xc_interface *xch, domid_t dom_id, int mode, + unsigned int size, uint64_t *buf); int xc_flask_op(xc_interface *xch, xen_flask_op_t *op); diff --git a/tools/libxc/xc_private.c b/tools/libxc/xc_private.c index f395594a8f..97663f219b 100644 --- a/tools/libxc/xc_private.c +++ b/tools/libxc/xc_private.c @@ -530,6 +530,28 @@ int xc_version(xc_interface *xch, int cmd, void *arg) return rc; } +int xc_edge_trace(xc_interface *xch, + domid_t dom_id, int mode, unsigned int size, uint64_t* arg) +{ + int rc; + + DECLARE_HYPERCALL_BOUNCE(arg, size * sizeof(uint64_t), + XC_HYPERCALL_BUFFER_BOUNCE_OUT); + + if ( xc_hypercall_bounce_pre(xch, arg) ) + { + PERROR("Could not bounce buffer for edge_trace hypercall"); + return -ENOMEM; + } + + rc = do_edge_trace(xch, dom_id, mode, size, HYPERCALL_BUFFER(arg)); + + xc_hypercall_bounce_post(xch, arg); + + return rc; +} + + unsigned long xc_make_page_below_4G( xc_interface *xch, uint32_t domid, unsigned long mfn) { diff --git a/tools/libxc/xc_private.h b/tools/libxc/xc_private.h index 1c27b0fded..60b0d8ebe3 100644 --- a/tools/libxc/xc_private.h +++ b/tools/libxc/xc_private.h @@ -229,6 +229,14 @@ static inline int do_xen_version(xc_interface *xch, int cmd, xc_hypercall_buffer cmd, HYPERCALL_BUFFER_AS_ARG(dest)); } +static inline int do_edge_trace(xc_interface *xch, domid_t dom_id, int mode, + unsigned int size, xc_hypercall_buffer_t *buf) +{ + DECLARE_HYPERCALL_BUFFER_ARGUMENT(buf); + return xencall4(xch->xcall, __HYPERVISOR_edge_trace, dom_id, mode, + size, HYPERCALL_BUFFER_AS_ARG(buf)); +} + static inline int do_physdev_op(xc_interface *xch, int cmd, void *op, size_t len) { int ret = -1; diff --git a/xen/Kconfig b/xen/Kconfig index 65d491d776..5ed2c9c390 100644 --- a/xen/Kconfig +++ b/xen/Kconfig @@ -38,4 +38,8 @@ config LTO If unsure, say N. +config TRACE_PC + bool "Enable tracing e.g. for fuzzing" + default false + source "Kconfig.debug" diff --git a/xen/Rules.mk b/xen/Rules.mk index 77bcd44922..dde14e3228 100644 --- a/xen/Rules.mk +++ b/xen/Rules.mk @@ -170,6 +170,10 @@ clean:: $(addprefix _clean_, $(subdir-all)) _clean_%/: FORCE $(MAKE) -f $(BASEDIR)/Rules.mk -C $* clean +ifeq ($(CONFIG_TRACE_PC),y) +$(objs-need-tracing): CFLAGS += -fsanitize-coverage=trace-pc +endif + %.o: %.c Makefile $(CC) $(CFLAGS) -c $< -o $@ diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c index c07999b518..a4d36517f9 100644 --- a/xen/arch/arm/traps.c +++ b/xen/arch/arm/traps.c @@ -1419,6 +1419,7 @@ static arm_hypercall_t arm_hypercall_table[] = { HYPERCALL(platform_op, 1), HYPERCALL_ARM(vcpu_op, 3), HYPERCALL(vm_assist, 2), + HYPERCALL(edge_trace, 4), }; #ifndef NDEBUG diff --git a/xen/arch/x86/hvm/hypercall.c b/xen/arch/x86/hvm/hypercall.c index e7238ce293..fed8363d8a 100644 --- a/xen/arch/x86/hvm/hypercall.c +++ b/xen/arch/x86/hvm/hypercall.c @@ -132,6 +132,7 @@ static const hypercall_table_t hvm_hypercall_table[] = { COMPAT_CALL(mmuext_op), HYPERCALL(xenpmu_op), COMPAT_CALL(dm_op), + HYPERCALL(edge_trace), HYPERCALL(arch_1) }; diff --git a/xen/arch/x86/hypercall.c b/xen/arch/x86/hypercall.c index e30181817a..caa9376d29 100644 --- a/xen/arch/x86/hypercall.c +++ b/xen/arch/x86/hypercall.c @@ -67,6 +67,7 @@ const hypercall_args_t hypercall_args_table[NR_hypercalls] = ARGS(tmem_op, 1), ARGS(xenpmu_op, 2), ARGS(dm_op, 3), + ARGS(edge_trace, 4), ARGS(mca, 1), ARGS(arch_1, 1), }; diff --git a/xen/arch/x86/pv/hypercall.c b/xen/arch/x86/pv/hypercall.c index 7c5e5a629d..24f931a895 100644 --- a/xen/arch/x86/pv/hypercall.c +++ b/xen/arch/x86/pv/hypercall.c @@ -79,6 +79,7 @@ static const hypercall_table_t pv_hypercall_table[] = { #endif HYPERCALL(xenpmu_op), COMPAT_CALL(dm_op), + HYPERCALL(edge_trace), HYPERCALL(mca), HYPERCALL(arch_1), }; diff --git a/xen/common/Makefile b/xen/common/Makefile index 26c5a64337..274baf084b 100644 --- a/xen/common/Makefile +++ b/xen/common/Makefile @@ -6,6 +6,8 @@ obj-y += cpupool.o obj-$(CONFIG_HAS_DEVICE_TREE) += device_tree.o obj-y += domctl.o obj-y += domain.o +obj-$(CONFIG_TRACE_PC) += edge_trace.o +obj-$(CONFIG_TRACE_PC) += edge_tracer.o obj-y += event_2l.o obj-y += event_channel.o obj-y += event_fifo.o @@ -80,3 +82,14 @@ subdir-$(CONFIG_GCOV) += gcov subdir-y += libelf subdir-$(CONFIG_HAS_DEVICE_TREE) += libfdt + +objs-need-tracing := bsearch.o \ + decompress.o device_tree.o domain.o domctl.o earlycpio.o grant_table.o \ + guestcopy.o gunzip.o inflate.o kernel.o kexec.o keyhandler.o kimage.o \ + lib.o livepatch.o lzo.o mem_access.o memory.o multicall.o notifier.o \ + page_alloc.o pdx.o perfc.o radix_tree.o rangeset.o \ + rbtree.o shutdown.o smp.o sort.o stop_machine.o string.o \ + symbols.o symbols-dummy.o sysctl.o time.o tmem.o \ + tmem_control.o tmem_xen.o trace.o unlz4.o unlzo.o unxz.o version.o \ + virtual_region.o vmap.o vm_event.o warning.o xenoprof.o \ + xmalloc_tlsf.o diff --git a/xen/common/edge_trace.c b/xen/common/edge_trace.c new file mode 100644 index 0000000000..7b325f4d17 --- /dev/null +++ b/xen/common/edge_trace.c @@ -0,0 +1,77 @@ +/****************************************************************************** + * edge_trace.c + * + * Copyright (c) 2017 Felix Schmoll + */ + +#include +#include +#include +#include + +long do_edge_trace(domid_t dom, int mode, unsigned int size, + XEN_GUEST_HANDLE_PARAM(uint64_t) buf) +{ + int ret = 0; + struct domain *d; + + if( dom == DOMID_SELF ) + d = current->domain; + else + d = get_domain_by_id(dom); + + if( !d ) + return -EINVAL; /* invalid domain */ + + switch ( mode ) + { + case TRACE_START: + { + if( d->tracing_buffer ) + { + ret = -EBUSY; /* domain already being traced */ + break; + } + + d->tracing_buffer_pos = 0; + d->tracing_buffer_size = size; + d->tracing_buffer = xmalloc_array(uint64_t, size); + + if( !d->tracing_buffer ) + ret = -ENOMEM; + break; + } + + case TRACE_STOP: + { + uint64_t* temp = d->tracing_buffer; + d->tracing_buffer = NULL; + + if( copy_to_guest(buf, temp, d->tracing_buffer_pos) ) + ret = -EFAULT; + + xfree(temp); + + ret = d->tracing_buffer_pos; + break; + } + + default: + ret = -ENOSYS; + } + + if( dom != DOMID_SELF ) + put_domain(d); + + return ret; +} + +/* + * Local variables: + * mode: C + * c-file-style: "BSD" + * c-basic-offset: 4 + * tab-width: 4 + * indent-tabs-mode: nil + * End: + */ diff --git a/xen/common/edge_tracer.c b/xen/common/edge_tracer.c new file mode 100644 index 0000000000..eb7ab96d8f --- /dev/null +++ b/xen/common/edge_tracer.c @@ -0,0 +1,25 @@ +/****************************************************************************** + * edge_tracer.c + * + * Copyright (c) 2017 Felix Schmoll + */ + +#include +#include + +void __sanitizer_cov_trace_pc(void) +{ + struct domain *d; + + if( system_state < SYS_STATE_active ) + return; + + d = current->domain; + + if(d->tracing_buffer && + d->tracing_buffer_pos < d->tracing_buffer_size) + { + d->tracing_buffer[d->tracing_buffer_pos++] = + (uint64_t) __builtin_return_address(0); + } +} diff --git a/xen/include/public/xen.h b/xen/include/public/xen.h index 2ac6b1e24d..0b113847c9 100644 --- a/xen/include/public/xen.h +++ b/xen/include/public/xen.h @@ -121,6 +121,7 @@ DEFINE_XEN_GUEST_HANDLE(xen_ulong_t); #define __HYPERVISOR_xc_reserved_op 39 /* reserved for XenClient */ #define __HYPERVISOR_xenpmu_op 40 #define __HYPERVISOR_dm_op 41 +#define __HYPERVISOR_edge_trace 42 /* Architecture-specific hypercall definitions. */ #define __HYPERVISOR_arch_0 48 diff --git a/xen/include/xen/edge_trace.h b/xen/include/xen/edge_trace.h new file mode 100644 index 0000000000..aa34c3ef96 --- /dev/null +++ b/xen/include/xen/edge_trace.h @@ -0,0 +1,19 @@ +/****************************************************************************** + * edge_trace.h + * + * Information structure that lives at the bottom of the per-cpu Xen stack. + */ + +#ifndef __TRACE_H__ +#define __TRACE_H__ + +#include +#include +#include + +#define TRACE_START 0 +#define TRACE_STOP 1 + +void __sanitizer_cov_trace_pc(void); + +#endif diff --git a/xen/include/xen/hypercall.h b/xen/include/xen/hypercall.h index cc99aea57d..48afc82bc0 100644 --- a/xen/include/xen/hypercall.h +++ b/xen/include/xen/hypercall.h @@ -83,6 +83,13 @@ do_xen_version( XEN_GUEST_HANDLE_PARAM(void) arg); extern long +do_edge_trace( + domid_t dom_id, + int mode, + unsigned int size, + XEN_GUEST_HANDLE_PARAM(uint64_t) buf); + +extern long do_console_io( int cmd, int count, diff --git a/xen/include/xen/sched.h b/xen/include/xen/sched.h index 1127ca99bd..eb0e9722d7 100644 --- a/xen/include/xen/sched.h +++ b/xen/include/xen/sched.h @@ -483,6 +483,12 @@ struct domain unsigned int guest_request_enabled : 1; unsigned int guest_request_sync : 1; } monitor; + +#ifdef CONFIG_TRACE_PC + uint64_t* tracing_buffer; + unsigned int tracing_buffer_pos; + unsigned int tracing_buffer_size; +#endif }; /* Protect updates/reads (resp.) of domain_list and domain_hash. */