tcmalloc issue

Commit Message

Chaitanya Huilgol March 27, 2015, 9:53 a.m. UTC

Hi,

Ubuntu Package version info: 
Package: libtcmalloc-minimal4
Versions: 
2.1-2ubuntu1 (/var/lib/apt/lists/in.archive.ubuntu.com_ubuntu_dists_trusty_main_binary-amd64_Packages)
 
On the OSD, increasing the thread caches from the default 32M has definitely helped, 
we have not tested this on client nodes though - probably will help there too. 

As a temporary non-intrusive workaround with the existing libtcmalloc-minimal4 library,
we are explicitly setting the thread cached values via rados-classes. (You can actually put this code into the OSD init path too) See patch below. 

Regards,
Chaitanya


From cafdbd41bb3492209716516910fa3a0e856ac7b8 Mon Sep 17 00:00:00 2001
From: Chaitanya Huilgol <[e-mail]>

Date: Fri, 27 Mar 2015 15:16:35 +0530
Subject: [PATCH] Set tcmalloc thread cache via rados-class

Workaround for bug in tcmalloc (2.1-2ubuntu1)which does not use the exported
thread cache size value via environment variable.  Using library init path
to explicitly set this value in tcmalloc. No actual object handlers are
registered and hence does not affect the data path.

Signed-off-by: Chaitanya Huilgol <[e-mail]>

---
 src/cls/Makefile.am                  |  5 ++++
 src/cls/tcmalloc_env/tcmalloc_env.cc | 52 ++++++++++++++++++++++++++++++++++++
 2 files changed, 57 insertions(+)
 create mode 100644 src/cls/tcmalloc_env/tcmalloc_env.cc

-- 
1.9.1

----Original Message-----
From: [e-mail] [mailto:[e-mail]] On Behalf Of Alexandre DERUMIER

Sent: Friday, March 27, 2015 12:52 PM
To: Somnath Roy
Cc: ceph-devel
Subject: Re: tcmalloc issue

Hi Sommath,

is it only on osds ?

or also clients ?

What is the tmalloc version on ubuntu ? (I would like to known if the problem exist also on debian)

Regards,

Alexandre
----- Mail original -----
De: "Somnath Roy" <[e-mail]>
À: "ceph-devel" <[e-mail]>
Envoyé: Vendredi 27 Mars 2015 00:18:57
Objet: tcmalloc issue

Sage,
Here is the tcmalloc issue I was talking in the performance meeting. 

http://code.google.com/p/gperftools/issues/detail?id=585 

The tcmalloc is consuming lot of cpus (and eventually slowing down the performance) with the following perf top trace. 

25.73% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::FetchFromSpans()
12.52% libtcmalloc.so.4.1.2 [.] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) 11.62% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseToSpans(void*)
1.56% [kernel] [k] __copy_user_nocache
1.44% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::RemoveRange(void*, void*, int) 1.39% libtcmalloc.so.4.1.2 [.] tcmalloc::CentralFreeList::ReleaseListToSpans(void*)
0.89% libtcmalloc.so.4.1.2 [.] operator new(unsigned long) 


The TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES is alleviating that but the env variable setting for this is a noop in the binary we are having in Ubuntu 14.04. It is fixed in gperftools-2.1.90 release. 
See the following link. 

http://code.google.com/p/gperftools/ 

We can request Ubuntu guys to incorporate this version or the latest tcmalloc version which is gperftools-2.4. 

It will be really helpful if you can push them to incorporate this in 14.04 updates. 

Thanks & Regards
Somnath 

________________________________ 

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to [e-mail] More majordomo info at http://vger.kernel.org/majordomo-info.html 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to [e-mail]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch

diff --git a/src/cls/Makefile.am b/src/cls/Makefile.am
index ea44fe7..abcf007 100644
--- a/src/cls/Makefile.am
+++ b/src/cls/Makefile.am
@@ -5,6 +5,11 @@  libcls_hello_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS)
 libcls_hello_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*'
 radoslib_LTLIBRARIES += libcls_hello.la
 
+libcls_tcmalloc_env_la_SOURCES = cls/tcmalloc_env/tcmalloc_env.cc
+libcls_tcmalloc_env_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS)
+libcls_tcmalloc_env_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*'
+radoslib_LTLIBRARIES += libcls_tcmalloc_env.la
+
 libcls_rbd_la_SOURCES = cls/rbd/cls_rbd.cc
 libcls_rbd_la_LIBADD = $(PTHREAD_LIBS) $(EXTRALIBS)
 libcls_rbd_la_LDFLAGS = ${AM_LDFLAGS} -module -avoid-version -shared -export-symbols-regex '.*__cls_.*'
diff --git a/src/cls/tcmalloc_env/tcmalloc_env.cc b/src/cls/tcmalloc_env/tcmalloc_env.cc
new file mode 100644
index 0000000..0907183
--- /dev/null
+++ b/src/cls/tcmalloc_env/tcmalloc_env.cc
@@ -0,0 +1,52 @@ 
+/*
+ * rados-classes based plugin to set TCmalloc environment variable
+ * 'TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES' as the existing tcmalloc
+ * library in ubuntu 14.04 LTS and centos 7.0 has a bug which does
+ * not honor this env
+ */ 
+#include <cstdlib>
+#ifdef HAVE_GPERFTOOLS_HEAP_PROFILER_H
+#include <gperftools/heap-profiler.h>
+#else
+#include <google/heap-profiler.h>
+#endif
+
+#ifdef HAVE_GPERFTOOLS_MALLOC_EXTENSION_H
+#include <gperftools/malloc_extension.h>
+#else
+#include <google/malloc_extension.h>
+#endif
+#include "objclass/objclass.h"
+
+CLS_VER (1, 0) CLS_NAME (tcmalloc_env)
+#define DEFAULT_CACHE_SIZE (32 * 1024 * 1024)
+
+void
+__cls_init ()
+{
+  size_t result;
+  size_t cache_sz;
+  char *env_cache_sz_str;
+
+  CLS_LOG (0, "TCMALLOC-ENV: Search");
+  env_cache_sz_str = getenv ("TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES");
+  if (env_cache_sz_str) {
+    cache_sz = strtoul (env_cache_sz_str, NULL, 0);
+    CLS_LOG(0, "TCMALLOC-ENV: Found: %lu\n", cache_sz);
+    if (cache_sz > DEFAULT_CACHE_SIZE) {
+      MallocExtension::instance ()->
+        SetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", cache_sz);
+      result = 0;
+      MallocExtension::instance ()->
+        GetNumericProperty ("tcmalloc.max_total_thread_cache_bytes", &result);
+      CLS_LOG (0, "TCMALLOC-ENV:Verify: max_total_thread_cache_bytes=%lu\n",
+        (unsigned long) result);
+    } else {
+      CLS_LOG (0, "TCMALLOC-ENV: Exported CacheSz=%lu <= Default=%lu - Ignored\n",
+        (unsigned long) cache_sz, (unsigned long) DEFAULT_CACHE_SIZE);
+    }
+  } else {
+      CLS_LOG(0, "TCMALLOC-ENV: Not Found\n");
+  }
+}
+/* EOF */