From patchwork Thu Jun 1 17:34:17 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dario Faggioli X-Patchwork-Id: 9760713 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 1D1D560363 for ; Thu, 1 Jun 2017 17:36:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 07CF827569 for ; Thu, 1 Jun 2017 17:36:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F024028518; Thu, 1 Jun 2017 17:36:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_MED,RCVD_IN_SORBS_SPAM,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0EEFA27569 for ; Thu, 1 Jun 2017 17:36:38 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dGTzK-00008F-Gu; Thu, 01 Jun 2017 17:34:22 +0000 Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dGTzJ-00007H-4e for xen-devel@lists.xenproject.org; Thu, 01 Jun 2017 17:34:21 +0000 Received: from [85.158.143.35] by server-9.bemta-6.messagelabs.com id DF/28-03557-C1050395; Thu, 01 Jun 2017 17:34:20 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrFIsWRWlGSWpSXmKPExsXiVRvkrCsdYBB p0LlXz+L7lslMDowehz9cYQlgjGLNzEvKr0hgzdjReI+5YEFNxfEJb1gaGA9GdTFycQgJzGCU WPdqMguIwyKwhlXi3cuTjCCOhMAlVomXp08CZTiBnDiJdw0fGCHsUokXC1+xgdhCAioSN7evY oIY9YNRYuLGFcwgCWEBPYkjR3+wQ9gWEicvXWYCsdkEDCTe7NjLCmKLCChJ3Fs1GayZWaCRSW LJtQ9gDSwCqhL7J+wEa+AV8JbY3bgLbBungK/EpFNnGCE2+0h8+zUBrEZUQE5i5eUWVoh6QYm TM58AXc0BNFRTYv0ufZAws4C8xPa3c5gnMIrMQlI1C6FqFpKqBYzMqxg1ilOLylKLdI0N9ZKK MtMzSnITM3N0DQ3M9HJTi4sT01NzEpOK9ZLzczcxAmOAAQh2MDYtCjzEKMnBpCTKu0DeIFKIL yk/pTIjsTgjvqg0J7X4EKMMB4eSBG+2H1BOsCg1PbUiLTMHGI0waQkOHiUR3hSQNG9xQWJucW Y6ROoUozHHhtXrvzBxTDqw/QuTEEtefl6qlDivAUipAEhpRmke3CBYkrjEKCslzMsIdJoQT0F qUW5mCar8K0ZxDkYlYd6VIFN4MvNK4Pa9AjqFCeiUF9v0QU4pSURISQHTjpR17eXvou/by+5u unyzoSlAVU1st3tx/5lcwwvHEtzf2ureTLx948PdxWar+XLaNk7K+35Bx/RRUkaWxacOQaG5y 3/VijMZdy1V4D0puG1CXWOJ8v2IA4o5hZxd/jJ67HNFXsULnHBNueV7wuaPXkfEaUH2oh09Vv K/5Zq1cw8lJPRmbFJiKc5INNRiLipOBADx88NiDQMAAA== X-Env-Sender: raistlin.df@gmail.com X-Msg-Ref: server-6.tower-21.messagelabs.com!1496338459!48822464!1 X-Originating-IP: [74.125.82.67] X-SpamReason: No, hits=0.5 required=7.0 tests=BODY_RANDOM_LONG X-StarScan-Received: X-StarScan-Version: 9.4.19; banners=-,-,- X-VirusChecked: Checked Received: (qmail 28890 invoked from network); 1 Jun 2017 17:34:19 -0000 Received: from mail-wm0-f67.google.com (HELO mail-wm0-f67.google.com) (74.125.82.67) by server-6.tower-21.messagelabs.com with AES128-GCM-SHA256 encrypted SMTP; 1 Jun 2017 17:34:19 -0000 Received: by mail-wm0-f67.google.com with SMTP id g15so13030255wmc.2 for ; Thu, 01 Jun 2017 10:34:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=XfLnSG/xDZOq7TitcTD559PsHDfuZQjR5orz4n8Y42E=; b=nj6faw9jVyogvhsvXA2CtaYQC1SM/PZEd1PKwbAZqTFIPL1f6dZm+PeSE4iJQzln7z RXLAc0m3Pb4RUm3RjYE7NKdTo0BoRjTqeifh+5Gc/P6zaOR3sVExZ6nBEXf2u46S8SC/ SJzPgyv5qaKAwxJUDbHhQwHr5Vo9ITtT4H4KMb5BpUkMoXOa5Xl0gPpul8a0VprneUAC gs8P/STlrpOVY/XO8MZXZiAlj4lNsuLQGVFoTLVbte5S0kPSXFHOK+GbjqnuKF1iqpEX eZi0WiEuoo7IbpMn9VOkB9RewAwJ/ehvB838nBCQR3eQfdjtti8JpoNMy+bhfJ8BMAo6 aERw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:from:to:cc:date:message-id :in-reply-to:references:user-agent:mime-version :content-transfer-encoding; bh=XfLnSG/xDZOq7TitcTD559PsHDfuZQjR5orz4n8Y42E=; b=CXP2JC9oWlQ7HIPmf+SylS2i6Y6virXhma9KSXM43gIKHweCSpxUUR0o8Yup2k3/4X scx5W+wOeXP+9NqLZjffz14rWCxuOAbd4FJ5oZ8VFNTOU83lyA8t0JBP6Rba5VBkqeox 5gKNAUu3nl0CxsLWo08ZpO7w6uTheAr2lSwN+lyqC8rN8ogyqaULzHhiF9QBlPA0D+zM QYRB+6umVz+irabSOUelUD9ctPZeLSEFe2JmvtEbgr7vaOPM2mZyMwfE5l18xAjdsP3a WFE/uCAXOnpOTnWkVNOqEk28NhNMkPkxjHj65b2BC1A7OWUEY8G/t97Ka8R5WN5xBD1+ 0sYg== X-Gm-Message-State: AODbwcCblUm4Qietj+kcL/39Lvgv/AU0f4fq1T0Xo9A3VFdR+XQovrI1 g8simuqWmNgydg== X-Received: by 10.28.22.71 with SMTP id 68mr194161wmw.67.1496338459336; Thu, 01 Jun 2017 10:34:19 -0700 (PDT) Received: from Solace.fritz.box ([80.66.223.25]) by smtp.gmail.com with ESMTPSA id b8sm22602502wrb.7.2017.06.01.10.34.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 01 Jun 2017 10:34:18 -0700 (PDT) From: Dario Faggioli To: xen-devel@lists.xenproject.org Date: Thu, 01 Jun 2017 19:34:17 +0200 Message-ID: <149633845700.12814.7130992212550379105.stgit@Solace.fritz.box> In-Reply-To: <149633614204.12814.14390287626133023934.stgit@Solace.fritz.box> References: <149633614204.12814.14390287626133023934.stgit@Solace.fritz.box> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Cc: Stefano Stabellini , George Dunlap , Andrew Cooper , Jennifer Herbert , Julien Grall , Jan Beulich Subject: [Xen-devel] [PATCH 06/15] xen: trace IRQ enabling/disabling X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Trace when interrupts are disabled and (re)enabled. Basically, we replace the IRQ disabling and enabling functions with helpers that does the same, but also output the proper trace record. For putting in the record something that will let us identify _where_ in the code (i.e., in what function) the IRQ manipulation is happening, use either: - current_text_addr(), - or __builtin_return_address(0). In fact, depending on whether the disabling/enabling happens in macros (like for local_irq_disable() and local_irq_enable()) or in actual functions (like in spin_lock_irq*()), it is either: - the actual content of the instruction pointer when IRQ are disabled/enabled, - or the return address of the utility function where IRQ are disabled/enabled, that will tell us what it is the actual piece of code that is asking for the IRQ manipulation operation. Gate this with its specific Kconfig option, and keep it in disabled state by default (i.e., don't build it, if not explicitly specified), as the impact on performance may be non negligible. Signed-off-by: Dario Faggioli Acked-by: George Dunlap --- Cc: George Dunlap Cc: Stefano Stabellini Cc: Julien Grall Cc: Jan Beulich Cc: Andrew Cooper Cc: Jennifer Herbert --- xen/Kconfig.debug | 11 ++++- xen/common/spinlock.c | 16 +++++-- xen/common/trace.c | 10 +++- xen/include/asm-arm/arm32/system.h | 12 +++++ xen/include/asm-arm/arm64/system.h | 12 +++++ xen/include/asm-x86/system.h | 85 ++++++++++++++++++++++++++++++++++-- xen/include/public/trace.h | 2 + xen/include/xen/rwlock.h | 33 +++++++++++--- 8 files changed, 161 insertions(+), 20 deletions(-) diff --git a/xen/Kconfig.debug b/xen/Kconfig.debug index 374c1c0..81910c9 100644 --- a/xen/Kconfig.debug +++ b/xen/Kconfig.debug @@ -98,7 +98,7 @@ config PERF_ARRAYS ---help--- Enables software performance counter array histograms. -config TRACING +menuconfig TRACING bool "Tracing" default y ---help--- @@ -106,6 +106,15 @@ config TRACING in per-CPU ring buffers. The 'xentrace' tool can be used to read the buffers and dump the content on the disk. +config TRACE_IRQSOFF + bool "Trace when IRQs are disabled and (re)enabled" if EXPERT = "y" + default n + depends on TRACING + ---help--- + Makes it possible to generate events _every_ time IRQs are disabled + and (re)enabled, with also an indication of where that happened. + Note that this comes with high overead and produces huge amount of + tracing data. config VERBOSE_DEBUG bool "Verbose debug messages" diff --git a/xen/common/spinlock.c b/xen/common/spinlock.c index 2a06406..33b903e 100644 --- a/xen/common/spinlock.c +++ b/xen/common/spinlock.c @@ -150,7 +150,9 @@ void _spin_lock(spinlock_t *lock) void _spin_lock_irq(spinlock_t *lock) { ASSERT(local_irq_is_enabled()); - local_irq_disable(); + _local_irq_disable(); + if ( unlikely(tb_init_done) ) + trace_irq_disable_ret(); _spin_lock(lock); } @@ -158,7 +160,9 @@ unsigned long _spin_lock_irqsave(spinlock_t *lock) { unsigned long flags; - local_irq_save(flags); + _local_irq_save(flags); + if ( unlikely(tb_init_done) ) + trace_irq_save_ret(flags); _spin_lock(lock); return flags; } @@ -175,13 +179,17 @@ void _spin_unlock(spinlock_t *lock) void _spin_unlock_irq(spinlock_t *lock) { _spin_unlock(lock); - local_irq_enable(); + if ( unlikely(tb_init_done) ) + trace_irq_enable_ret(); + _local_irq_enable(); } void _spin_unlock_irqrestore(spinlock_t *lock, unsigned long flags) { _spin_unlock(lock); - local_irq_restore(flags); + if ( unlikely(tb_init_done) ) + trace_irq_restore_ret(flags); + _local_irq_restore(flags); } int _spin_is_locked(spinlock_t *lock) diff --git a/xen/common/trace.c b/xen/common/trace.c index 2c18462..71202df 100644 --- a/xen/common/trace.c +++ b/xen/common/trace.c @@ -722,7 +722,12 @@ void __trace_var(u32 event, bool_t cycles, unsigned int extra, /* Read tb_init_done /before/ t_bufs. */ smp_rmb(); - spin_lock_irqsave(&this_cpu(t_lock), flags); + /* + * spin_lock_irqsave() would call local_irq_save(), which (may) + * call __trace_var(). Open code it to avoid recursing. + */ + _local_irq_save(flags); + spin_lock(&this_cpu(t_lock)); buf = this_cpu(t_bufs); @@ -809,7 +814,8 @@ void __trace_var(u32 event, bool_t cycles, unsigned int extra, __insert_record(buf, event, extra, cycles, rec_size, extra_data); unlock: - spin_unlock_irqrestore(&this_cpu(t_lock), flags); + spin_unlock(&this_cpu(t_lock)); + _local_irq_restore(flags); /* Notify trace buffer consumer that we've crossed the high water mark. */ if ( likely(buf!=NULL) diff --git a/xen/include/asm-arm/arm32/system.h b/xen/include/asm-arm/arm32/system.h index c617b40..20871ad 100644 --- a/xen/include/asm-arm/arm32/system.h +++ b/xen/include/asm-arm/arm32/system.h @@ -4,6 +4,8 @@ #include +#include + #define local_irq_disable() asm volatile ( "cpsid i @ local_irq_disable\n" : : : "cc" ) #define local_irq_enable() asm volatile ( "cpsie i @ local_irq_enable\n" : : : "cc" ) @@ -41,6 +43,16 @@ static inline int local_irq_is_enabled(void) #define local_abort_enable() __asm__("cpsie a @ __sta\n" : : : "memory", "cc") #define local_abort_disable() __asm__("cpsid a @ __sta\n" : : : "memory", "cc") +/* We do not support tracing (at all) yet */ +#define trace_irq_disable_ret() do { } while ( 0 ) +#define trace_irq_enable_ret() do { } while ( 0 ) +#define trace_irq_save_ret(_x) do { } while ( 0 ) +#define trace_irq_restore_ret(_x) do { } while ( 0 ) +#define _local_irq_disable() local_irq_disable() +#define _local_irq_enable() local_irq_enable() +#define _local_irq_save(_x) local_irq_save(_x) +#define _local_irq_restore(_x) local_irq_restore(_x) + static inline int local_fiq_is_enabled(void) { unsigned long flags; diff --git a/xen/include/asm-arm/arm64/system.h b/xen/include/asm-arm/arm64/system.h index 2e2ee21..6603b0c 100644 --- a/xen/include/asm-arm/arm64/system.h +++ b/xen/include/asm-arm/arm64/system.h @@ -4,6 +4,8 @@ #include +#include + /* Uses uimm4 as a bitmask to select the clearing of one or more of * the DAIF exception mask bits: * bit 3 selects the D mask, @@ -44,6 +46,16 @@ : "memory"); \ }) +/* We do not support tracing (at all) yet */ +#define trace_irq_disable_ret() do { } while ( 0 ) +#define trace_irq_enable_ret() do { } while ( 0 ) +#define trace_irq_save_ret(_x) do { } while ( 0 ) +#define trace_irq_restore_ret(_x) do { } while ( 0 ) +#define _local_irq_disable() local_irq_disable() +#define _local_irq_enable() local_irq_enable() +#define _local_irq_save(_x) local_irq_save(_x) +#define _local_irq_restore(_x) local_irq_restore(_x) + static inline int local_irq_is_enabled(void) { unsigned long flags; diff --git a/xen/include/asm-x86/system.h b/xen/include/asm-x86/system.h index eb498f5..0e7bf01 100644 --- a/xen/include/asm-x86/system.h +++ b/xen/include/asm-x86/system.h @@ -5,6 +5,8 @@ #include #include +#include + #define read_sreg(name) \ ({ unsigned int __sel; \ asm volatile ( "mov %%" STR(name) ",%0" : "=r" (__sel) ); \ @@ -185,8 +187,8 @@ static always_inline unsigned long __xadd( #define set_mb(var, value) do { xchg(&var, value); } while (0) #define set_wmb(var, value) do { var = value; wmb(); } while (0) -#define local_irq_disable() asm volatile ( "cli" : : : "memory" ) -#define local_irq_enable() asm volatile ( "sti" : : : "memory" ) +#define _local_irq_disable() asm volatile ( "cli" : : : "memory" ) +#define _local_irq_enable() asm volatile ( "sti" : : : "memory" ) /* used in the idle loop; sti takes one instruction cycle to complete */ #define safe_halt() asm volatile ( "sti; hlt" : : : "memory" ) @@ -198,12 +200,12 @@ static always_inline unsigned long __xadd( BUILD_BUG_ON(sizeof(x) != sizeof(long)); \ asm volatile ( "pushf" __OS " ; pop" __OS " %0" : "=g" (x)); \ }) -#define local_irq_save(x) \ +#define _local_irq_save(x) \ ({ \ local_save_flags(x); \ - local_irq_disable(); \ + _local_irq_disable(); \ }) -#define local_irq_restore(x) \ +#define _local_irq_restore(x) \ ({ \ BUILD_BUG_ON(sizeof(x) != sizeof(long)); \ asm volatile ( "pushfq\n\t" \ @@ -214,6 +216,79 @@ static always_inline unsigned long __xadd( "ri" ( (x) & X86_EFLAGS_IF ) ); \ }) +#ifdef CONFIG_TRACE_IRQSOFF + +#define TRACE_LOCAL_ADDR ((uint64_t) current_text_addr()) +#define TRACE_RET_ADDR ((uint64_t) __builtin_return_address(0)) + +#define trace_irq_disable(_a) \ +({ \ + uint64_t addr = _a; \ + __trace_var(TRC_HW_IRQ_DISABLE, 1, sizeof(addr), &addr); \ +}) +#define trace_irq_enable(_a) \ +({ \ + uint64_t addr = _a; \ + __trace_var(TRC_HW_IRQ_ENABLE, 1, sizeof(addr), &addr); \ +}) +#define trace_irq_save(_x, _a) \ +({ \ + uint64_t addr = _a; \ + if ( _x & X86_EFLAGS_IF ) \ + __trace_var(TRC_HW_IRQ_DISABLE, 1, sizeof(addr), &addr); \ +}) +#define trace_irq_restore(_x, _a) \ +({ \ + uint64_t addr = _a; \ + if ( _x & X86_EFLAGS_IF ) \ + __trace_var(TRC_HW_IRQ_ENABLE, 1, sizeof(addr), &addr); \ +}) + +#define trace_irq_disable_ret() trace_irq_disable(TRACE_RET_ADDR) +#define trace_irq_enable_ret() trace_irq_enable(TRACE_RET_ADDR) +#define trace_irq_save_ret(_x) trace_irq_save(_x, TRACE_RET_ADDR) +#define trace_irq_restore_ret(_x) trace_irq_restore(_x, TRACE_RET_ADDR) + +#define local_irq_disable() \ +({ \ + bool_t irqon = local_irq_is_enabled(); \ + _local_irq_disable(); \ + if ( unlikely(tb_init_done && irqon) ) \ + trace_irq_disable(TRACE_LOCAL_ADDR); \ +}) + +#define local_irq_enable() \ +({ \ + if ( unlikely(tb_init_done) ) \ + trace_irq_enable(TRACE_LOCAL_ADDR); \ + _local_irq_enable(); \ +}) + +#define local_irq_save(_x) \ +({ \ + local_save_flags(_x); \ + _local_irq_disable(); \ + if ( unlikely(tb_init_done) ) \ + trace_irq_save(_x, TRACE_LOCAL_ADDR); \ +}) + +#define local_irq_restore(_x) \ +({ \ + if ( unlikely(tb_init_done) ) \ + trace_irq_restore(_x, TRACE_LOCAL_ADDR); \ + _local_irq_restore(_x); \ +}) +#else /* !TRACE_IRQSOFF */ +#define trace_irq_disable_ret() do { } while ( 0 ) +#define trace_irq_enable_ret() do { } while ( 0 ) +#define trace_irq_save_ret(_x) do { } while ( 0 ) +#define trace_irq_restore_ret(_x) do { } while ( 0 ) +#define local_irq_disable() _local_irq_disable() +#define local_irq_enable() _local_irq_enable() +#define local_irq_save(_x) _local_irq_save(_x) +#define local_irq_restore(_x) _local_irq_restore(_x) +#endif /* TRACE_IRQSOFF */ + static inline int local_irq_is_enabled(void) { unsigned long flags; diff --git a/xen/include/public/trace.h b/xen/include/public/trace.h index f66a7af..1692a79 100644 --- a/xen/include/public/trace.h +++ b/xen/include/public/trace.h @@ -275,6 +275,8 @@ #define TRC_HW_IRQ_ENTER (TRC_HW_IRQ + 0xA) #define TRC_HW_IRQ_GUEST (TRC_HW_IRQ + 0xB) #define TRC_HW_IRQ_EXIT (TRC_HW_IRQ + 0xC) +#define TRC_HW_IRQ_DISABLE (TRC_HW_IRQ + 0xD) +#define TRC_HW_IRQ_ENABLE (TRC_HW_IRQ + 0xE) /* * Event Flags diff --git a/xen/include/xen/rwlock.h b/xen/include/xen/rwlock.h index 35657c5..04f50e5 100644 --- a/xen/include/xen/rwlock.h +++ b/xen/include/xen/rwlock.h @@ -73,14 +73,19 @@ static inline void _read_lock(rwlock_t *lock) static inline void _read_lock_irq(rwlock_t *lock) { ASSERT(local_irq_is_enabled()); - local_irq_disable(); + _local_irq_disable(); + if ( unlikely(tb_init_done) ) + trace_irq_disable_ret(); _read_lock(lock); } static inline unsigned long _read_lock_irqsave(rwlock_t *lock) { unsigned long flags; - local_irq_save(flags); + + _local_irq_save(flags); + if ( unlikely(tb_init_done) ) + trace_irq_save_ret(flags); _read_lock(lock); return flags; } @@ -100,13 +105,17 @@ static inline void _read_unlock(rwlock_t *lock) static inline void _read_unlock_irq(rwlock_t *lock) { _read_unlock(lock); - local_irq_enable(); + if ( unlikely(tb_init_done) ) + trace_irq_enable_ret(); + _local_irq_enable(); } static inline void _read_unlock_irqrestore(rwlock_t *lock, unsigned long flags) { _read_unlock(lock); - local_irq_restore(flags); + if ( unlikely(tb_init_done) ) + trace_irq_restore_ret(flags); + _local_irq_restore(flags); } static inline int _rw_is_locked(rwlock_t *lock) @@ -130,7 +139,9 @@ static inline void _write_lock(rwlock_t *lock) static inline void _write_lock_irq(rwlock_t *lock) { ASSERT(local_irq_is_enabled()); - local_irq_disable(); + _local_irq_disable(); + if ( unlikely(tb_init_done) ) + trace_irq_disable_ret(); _write_lock(lock); } @@ -138,7 +149,9 @@ static inline unsigned long _write_lock_irqsave(rwlock_t *lock) { unsigned long flags; - local_irq_save(flags); + _local_irq_save(flags); + if ( unlikely(tb_init_done) ) + trace_irq_save_ret(flags); _write_lock(lock); return flags; } @@ -171,13 +184,17 @@ static inline void _write_unlock(rwlock_t *lock) static inline void _write_unlock_irq(rwlock_t *lock) { _write_unlock(lock); - local_irq_enable(); + if ( unlikely(tb_init_done) ) + trace_irq_enable_ret(); + _local_irq_enable(); } static inline void _write_unlock_irqrestore(rwlock_t *lock, unsigned long flags) { _write_unlock(lock); - local_irq_restore(flags); + if ( unlikely(tb_init_done) ) + trace_irq_restore_ret(flags); + _local_irq_restore(flags); } static inline int _rw_is_write_locked(rwlock_t *lock)