From patchwork Fri Mar 27 14:37:35 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alexandre DERUMIER X-Patchwork-Id: 6107891 Return-Path: X-Original-To: patchwork-ceph-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 171FC9F2A9 for ; Fri, 27 Mar 2015 14:37:48 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id D25AA203DF for ; Fri, 27 Mar 2015 14:37:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 401CC203FB for ; Fri, 27 Mar 2015 14:37:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932108AbbC0Ohj (ORCPT ); Fri, 27 Mar 2015 10:37:39 -0400 Received: from mailpro.odiso.net ([89.248.209.98]:47722 "EHLO mailpro.odiso.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932097AbbC0Ohh convert rfc822-to-8bit (ORCPT ); Fri, 27 Mar 2015 10:37:37 -0400 Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 958DC40B0E79F; Fri, 27 Mar 2015 15:37:35 +0100 (CET) Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id oxOXTX-zJ5BA; Fri, 27 Mar 2015 15:37:35 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by mailpro.odiso.net (Postfix) with ESMTP id 6DB5D4091A615; Fri, 27 Mar 2015 15:37:35 +0100 (CET) X-Virus-Scanned: amavisd-new at mailpro.odiso.com Received: from mailpro.odiso.net ([127.0.0.1]) by localhost (mailpro.odiso.net [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id dyu9lypjC0im; Fri, 27 Mar 2015 15:37:35 +0100 (CET) Received: from mailpro.odiso.net (mailpro.odiso.net [10.1.31.112]) by mailpro.odiso.net (Postfix) with ESMTP id 44AF14042D2D9; Fri, 27 Mar 2015 15:37:35 +0100 (CET) Date: Fri, 27 Mar 2015 15:37:35 +0100 (CET) From: Alexandre DERUMIER To: Chaitanya Huilgol Cc: Somnath Roy , ceph-devel Message-ID: <250283338.1321814.1427467055129.JavaMail.zimbra@oxygem.tv> In-Reply-To: <9E914F5BD7F48A4782456CEB550A42281296767B@SACMBXIP01.sdcorp.global.sandisk.com> References: <755F6B91B3BE364F9BCA11EA3F9E0C6F28CAD14A@SACMBXIP01.sdcorp.global.sandisk.com> <976582620.219210.1427440913479.JavaMail.zimbra@oxygem.tv> <9E914F5BD7F48A4782456CEB550A42281296767B@SACMBXIP01.sdcorp.global.sandisk.com> Subject: Re: tcmalloc issue MIME-Version: 1.0 X-Mailer: Zimbra 8.6.0_GA_1153 (ZimbraWebClient - GC38 (Linux)/8.6.0_GA_1153) Thread-Topic: tcmalloc issue Thread-Index: AdBoGzqknAhM+uDVSGmEvUfc+7XOztEO27E90Q462gCDGrUDMg== Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP >>Ubuntu Package version info: >>Package: libtcmalloc-minimal4 >>Versions: >>2.1-2ubuntu1 (/var/lib/apt/lists/in.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages) Thanks ! looking at debian, debian wheezy : libtcmalloc-minimal4 (2.0-2) Seem to have the bug (I have looked inside debian src package) debian jessie : libtcmalloc-minimal4 (2.2.1-0.2) seem to be ok >>we have not tested this on client nodes though - probably will help there too. I ask about client side, because from my debian client previous bench, I have 4x more cpu on client librbd vs krbd, so maybe it's coming from libtcmalloc ----- Mail original ----- De: "Chaitanya Huilgol" À: "aderumier" , "Somnath Roy" Cc: "ceph-devel" Envoyé: Vendredi 27 Mars 2015 10:53:30 Objet: RE: tcmalloc issue Hi, Ubuntu Package version info: Package: libtcmalloc-minimal4 Versions: 2.1-2ubuntu1 (/var/lib/apt/lists/in.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages) On the OSD, increasing the thread caches from the default 32M has definitely helped, we have not tested this on client nodes though - probably will help there too. As a temporary non-intrusive workaround with the existing libtcmalloc-minimal4 library, we are explicitly setting the thread cached values via rados-classes. (You can actually put this code into the OSD init path too) See patch below. Regards, Chaitanya From cafdbd41bb3492209716516910fa3a0e856ac7b8 Mon Sep 17 00:00:00 2001 From: Chaitanya Huilgol Date: Fri, 27 Mar 2015 15:16:35 +0530 Subject: [PATCH] Set tcmalloc thread cache via rados-class Workaround for bug in tcmalloc (2.1-2ubuntu1)which does not use the exported thread cache size value via environment variable. Using library init path to explicitly set this value in tcmalloc. No actual object handlers are registered and hence does not affect the data path. Signed-off-by: Chaitanya Huilgol --- src/cls/Makefile.am | 5 ++++ src/cls/tcmalloc_env/tcmalloc_env.cc | 52 ++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) create mode 100644 src/cls/tcmalloc_env/tcmalloc_env.cc diff --git a/src/cls/Makefile.am b/src/cls/Makefile.am index ea44fe7..abcf007 100644 --- a/src/cls/Makefile.am +++ b/src/cls/Makefile.am @@ -5,6 +5,11 @@ libcls_hello_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) libcls_hello_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*' radoslib_LTLIBRARIES += libcls_hello.la +libcls_tcmalloc_env_la_SOURCES = cls/tcmalloc_env/tcmalloc_env.cc +libcls_tcmalloc_env_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) +libcls_tcmalloc_env_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*' +radoslib_LTLIBRARIES += libcls_tcmalloc_env.la + libcls_rbd_la_SOURCES = cls/rbd/cls_rbd.cc libcls_rbd_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS) libcls_rbd_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*' diff --git a/src/cls/tcmalloc_env/tcmalloc_env.cc b/src/cls/tcmalloc_env/tcmalloc_env.cc new file mode 100644 index 0000000..0907183 --- /dev/null +++ b/src/cls/tcmalloc_env/tcmalloc_env.cc @@ -0,0 +1,52 @@ +/* + * rados-classes based plugin to set TCmalloc environment variable + * 'TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES' as the existing tcmalloc + * library in ubuntu 14.04 LTS and centos 7.0 has a bug which does + * not honor this env + */ +#include +#ifdef HAVE_GPERFTOOLS_HEAP_PROFILER_H +#include +#else +#include +#endif + +#ifdef HAVE_GPERFTOOLS_MALLOC_EXTENSION_H +#include +#else +#include +#endif +#include "objclass/objclass.h" + +CLS_VER (1, 0) CLS_NAME (tcmalloc_env) +#define DEFAULT_CACHE_SIZE (32 * 1024 * 1024) + +void +__cls_init () +{ + size_t result; + size_t cache_sz; + char *env_cache_sz_str; + + CLS_LOG (0, "TCMALLOC-ENV: Search"); + env_cache_sz_str = getenv ("TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES"); + if (env_cache_sz_str) { + cache_sz = strtoul (env_cache_sz_str, NULL, 0); + CLS_LOG(0, "TCMALLOC-ENV: Found: %lu\n", cache_sz); + if (cache_sz > DEFAULT_CACHE_SIZE) { + MallocExtension::instance ()-> + SetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", cache_sz); + result = 0; + MallocExtension::instance ()-> + GetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", &result); + CLS_LOG (0, "TCMALLOC-ENV:Verify: max_total_thread_cache_bytes=%lu\n", + (unsigned long) result); + } else { + CLS_LOG (0, "TCMALLOC-ENV: Exported CacheSz=%lu <= Default=%lu - Ignored\n", + (unsigned long) cache_sz, (unsigned long) DEFAULT_CACHE_SIZE); + } + } else { + CLS_LOG(0, "TCMALLOC-ENV: Not Found\n"); + } +} +/* EOF */