tcmalloc issue

Commit Message

Alexandre DERUMIER March 27, 2015, 2:37 p.m. UTC

>>Ubuntu Package version info: 
>>Package: libtcmalloc-minimal4 
>>Versions: 
>>2.1-2ubuntu1 (/var/lib/apt/lists/in.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages) 

Thanks !

looking at debian,
debian wheezy : libtcmalloc-minimal4 (2.0-2)

Seem to have the bug (I have looked inside debian src package)

debian jessie : libtcmalloc-minimal4 (2.2.1-0.2)
seem to be ok



>>we have not tested this on client nodes though - probably will help there too. 

I ask about client side, because from my debian client previous bench, 
I have 4x more cpu on client librbd vs krbd, so maybe it's coming from libtcmalloc



----- Mail original -----
De: "Chaitanya Huilgol" <[e-mail]>
À: "aderumier" <[e-mail]>, "Somnath Roy" <[e-mail]>
Cc: "ceph-devel" <[e-mail]>
Envoyé: Vendredi 27 Mars 2015 10:53:30
Objet: RE: tcmalloc issue

Hi, 

Ubuntu Package version info: 
Package: libtcmalloc-minimal4 
Versions: 
2.1-2ubuntu1 (/var/lib/apt/lists/in.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages) 

On the OSD, increasing the thread caches from the default 32M has definitely helped, 
we have not tested this on client nodes though - probably will help there too. 

As a temporary non-intrusive workaround with the existing libtcmalloc-minimal4 library, 
we are explicitly setting the thread cached values via rados-classes. (You can actually put this code into the OSD init path too) See patch below. 

Regards, 
Chaitanya 


From cafdbd41bb3492209716516910fa3a0e856ac7b8 Mon Sep 17 00:00:00 2001 
From: Chaitanya Huilgol <[e-mail]> 
Date: Fri, 27 Mar 2015 15:16:35 +0530 
Subject: [PATCH] Set tcmalloc thread cache via rados-class 

Workaround for bug in tcmalloc (2.1-2ubuntu1)which does not use the exported 
thread cache size value via environment variable. Using library init path 
to explicitly set this value in tcmalloc. No actual object handlers are 
registered and hence does not affect the data path. 

Signed-off-by: Chaitanya Huilgol <[e-mail]> 
--- 
src/cls/Makefile.am | 5 ++++ 
src/cls/tcmalloc_env/tcmalloc_env.cc | 52 ++++++++++++++++++++++++++++++++++++ 
2 files changed, 57 insertions(+) 
create mode 100644 src/cls/tcmalloc_env/tcmalloc_env.cc

Patch

diff --git a/src/cls/Makefile.am b/src/cls/Makefile.am 
index ea44fe7..abcf007 100644 
--- a/src/cls/Makefile.am 
+++ b/src/cls/Makefile.am 
@@ -5,6 +5,11 @@  libcls_hello_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) 
libcls_hello_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*' 
radoslib_LTLIBRARIES += libcls_hello.la 

+libcls_tcmalloc_env_la_SOURCES = cls/tcmalloc_env/tcmalloc_env.cc 
+libcls_tcmalloc_env_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) 
+libcls_tcmalloc_env_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*' 
+radoslib_LTLIBRARIES += libcls_tcmalloc_env.la 
+ 
libcls_rbd_la_SOURCES = cls/rbd/cls_rbd.cc 
libcls_rbd_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) 
libcls_rbd_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*' 
diff --git a/src/cls/tcmalloc_env/tcmalloc_env.cc b/src/cls/tcmalloc_env/tcmalloc_env.cc 
new file mode 100644 
index 0000000..0907183 
--- /dev/null 
+++ b/src/cls/tcmalloc_env/tcmalloc_env.cc 
@@ -0,0 +1,52 @@  
+/* 
+ * rados-classes based plugin to set TCmalloc environment variable 
+ * 'TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES' as the existing tcmalloc 
+ * library in ubuntu 14.04 LTS and centos 7.0 has a bug which does 
+ * not honor this env 
+ */ 
+#include <cstdlib> 
+#ifdef HAVE_GPERFTOOLS_HEAP_PROFILER_H 
+#include <gperftools/heap-profiler.h> 
+#else 
+#include <google/heap-profiler.h> 
+#endif 
+ 
+#ifdef HAVE_GPERFTOOLS_MALLOC_EXTENSION_H 
+#include <gperftools/malloc_extension.h> 
+#else 
+#include <google/malloc_extension.h> 
+#endif 
+#include "objclass/objclass.h" 
+ 
+CLS_VER (1, 0) CLS_NAME (tcmalloc_env) 
+#define DEFAULT_CACHE_SIZE (32 * 1024 * 1024) 
+ 
+void 
+__cls_init () 
+{ 
+ size_t result; 
+ size_t cache_sz; 
+ char *env_cache_sz_str; 
+ 
+ CLS_LOG (0, "TCMALLOC-ENV: Search"); 
+ env_cache_sz_str = getenv ("TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES"); 
+ if (env_cache_sz_str) { 
+ cache_sz = strtoul (env_cache_sz_str, NULL, 0); 
+ CLS_LOG(0, "TCMALLOC-ENV: Found: %lu\n", cache_sz); 
+ if (cache_sz > DEFAULT_CACHE_SIZE) { 
+ MallocExtension::instance ()-> 
+ SetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", cache_sz); 
+ result = 0; 
+ MallocExtension::instance ()-> 
+ GetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", &result); 
+ CLS_LOG (0, "TCMALLOC-ENV:Verify: max_total_thread_cache_bytes=%lu\n", 
+ (unsigned long) result); 
+ } else { 
+ CLS_LOG (0, "TCMALLOC-ENV: Exported CacheSz=%lu <= Default=%lu - Ignored\n", 
+ (unsigned long) cache_sz, (unsigned long) DEFAULT_CACHE_SIZE); 
+ } 
+ } else { 
+ CLS_LOG(0, "TCMALLOC-ENV: Not Found\n"); 
+ } 
+} 
+/* EOF */